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(57) An Image processing device that reduces the 
memory size of image data inputted by scanner, digital 
camera or the like or sent in via communication means 
tike the Intemet and which outputs a clear enlarged im- 
age. Rrst, for reduction of the memory size of Image da- 
ta, low component extracting means extracts the low fre- 
quency components of image data, and the relation in- 
fomiation between high components and the low fre- 
quency components is obtained and encoded. To output 
the enlarged image clearly, Enlarged Frequency esti- 
mating means estimates the frequency components re- 
quired for enlargement of an image by edge extraction 
and nonlinear interpolation method. Furthermore, im- 
age enlarging means estimates the enlarged image by 
Wavelet transfomi. 
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Description 

[0001] The present invention relates to an image processing device and image processing method for reducing the 
memory size of image data inputted by the scanner, digital camera etc. or image data sent through communication 

5 means such as the Internet and outputting those data clearly 

[0002] With technical innovation in recent years, electronic photographing apparatuses have come to be put to prac- 
tical use that memorize Images on memory media such as magnetic media or semiconductor memory Instead of silver 
film. Especially, the digital camera that reads out the density and color using the charge coupled devices (CCD) element 
and memorizes them on a semiconductor memory or the lilce has found its way into homes because of the handiness 

10 and low price. The resolution of the CCD element is being upgraded from 1 ,000,000 picture elements to 2,000,000 
picture elements. Depending on the number of effective picture elements, the CCD element has a resolution capacity 
of approximately 1 ,280 picture elements x 980 picture elements, and the Image data size Is very large and it Is necessary 
to compress the image data. For the still image compression, the JPEG (Joint Photographic Coding Experts Group) 
standard is generally adopted. The prior art image processing device for data compression by this JPEG method Is 

15 shown in a block diagram in FIG. 39. 

[0003] An original image is read by an image Input means 1 0 formed of CCD elements etc. On the original image, 
lossy compressbn (non-reversible compression) by the JPEG is carried out by compression means 3900. 
[0004] That is, discrete cosine transfomn of the original image is perfomied by discrete cosine transfonn (DCT) means 
3906 whereby the original image is transfonned into signals in the frequency space, and the obtained transfonn coef- 

20 fident is quantized by quantization means 3907 using a quantization table 3901 . The results of this quantization are 
transfonned Into a code string by entropy encoding means 3908 on the basis of an entropy encoding table 3902, and 
this code string is stored on a storage medium 15. This processing Is continued until after the compression of all the 
original images is over. In this connection, a lossless compression (reversible compression) method In which an image 
can be restored without distortion is also proposed for the JPEG standard. When the lossress compression is used, 

25 the compression ratio, which is defined as the ratio of original image size to the compressed image size, is very low. 
Therfore, lossy compression is generally used. But when lossy compression is used, the original image are not exactly 
reconstmcted because of the quantization enror as In FIG. 39 and the rounding en-or by DCT. Of the two reversible 
factors, the quantization error has bad influence on the quality of the reconstructed image especially. 
[0005] Meanwhile, image processing devices such as the digital camera are generally provided with a display unit 

30 such as liquid crystal display to review a photographed image right there, to retrieve the image for data editing etc. 
Then, the resolution of CCD is high as compared with the resolution of the display unit for review, and when an com- 
pressed image data stored on the storage medium 15 Is displayed on those display units, this compressed image data 
Is expanded and the thinning out of picture elements is conducted to transform the resolution. 
[0006] First, expanding means 3903 In FIG. 39 expands a compressed image data stored on the storage medium 

35 1 5. In this process, the coefficient of entropy-encoded image signal is brought back to the quantized transform coeffi- 
cients by entropy decoding means 3911 and DCT coefficient by inverse quantization means 3910. Then, on this DCT 
coeffk^ient, inverse discrete cosine transfomnlng (IDCT) Is perfonned by IDCT means 3909 to expand the data to the 
value of the original image data space. This expanded image data is outputted on the printer, CFH" etc. by Original 
Image (01) output means 3912 in accordance with an instruction signal inputted by the user through the mouse or the 

40 like (not shown). 

[0007] On the other hand, means 3904 for thinning out images thins out picture elements from the all image data 
expanded by expanding rrieans 3903 according to the resolution at means 3905 for displaying reduced image for review 
display, and this thinned out image data is displayed for review by means 3905 for displaying reduced image. 
[0008] Now, to think of reading an original data of JiS paper size A4, for example, using an digital camera having 
45 CCD with 2,000,000 picture elements, it is clear that the digital camera has a resolution of only some 1 00 (dots/inch) 
if the resolution is estimated from the number of pbture elements of the CCD element, though that is not always 
applicable because of the focus distance and the like. This resolution is far more coarse than that in an image obtained 
on the silver film, and the expression is still insuffk:ient with regard to the edge information and clearness of details in 
a natural image. 

50 [0009] For this reason, when an image inputted by the digital camera is outputted on high-resolution printers like the 
laser printer and ink jet printer or is displayed on a high-resolution display connected to the computer for data editing, 
the low-resolution image has to be transformed into a high-resolution data. In recent years, in this connection, the 
sending of image data by communk:ation means such as the Internet has been getting popular, but since the commu- 
nication speed and the data size that can be sent at a time are limited, image data are lowered in resolution before 

55 being sent in many cases. Because of that, the image data sent tends to become an image with blurred details when 
displayed on the receiver's side. And there is a mounting call among the users that low-resolution images sent by the 
internet etc. should be upgraded in resolution. It is noted that high resolution transfomn and enlargement of Image are 
synonymous, and there will be explained the conventional methods of enlarging an image with precision. 
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[0010] The conventional methods of enlarging an image are roughly classified into the following two techniques: 

(1) Merely interpolating between the picture elements. 

(2) Enlarging an image after transfonning the image data in the real space into an image data in the frequency 
5 space using the orthogonal transfonm techniques such as FFT (fast Fourier transform) or DCT as described in 

unexamined Japanese patent applications 2-76472 and 5-1 67920. 

[0011] First, a conventional system example to realize the technique of interpolating between the picture elements 
in (1) is shown in a block diagram In FIG. 42. On an original image obtained by Image input means 10, the picture 
10 elements are linear interpolated between the picture elements in accordance with the following Fomnula 1 . In formula 
1 . Da is picture element data at point A, Db is pic^re element data at point B, Dc is picture element data at point C, 
and Dd is picture element data at point D. De is picture element data at point E to be woriced out. 

Fomnula 1 

15 

[0012] 

De = (1 -\i)x(1 -v)xDa + \ix(1 'V)xDb 

20 

-h(1-\L)xvxDc + \LXvxDd 

[001 3] Since an Interpolated image data tends to be an averaged one and to blur in edge etc. , the interpolated picture 
element position and interpolated data using edge infomnation extracted from the original image is often adjusted. In 

25 FIG. 42, edge information is extracted by edge extracting means 4200, and is enlarged to a desired enlargement ratio 
by edge interpolation means 4201 to obtain an enlarged edge. And the interpolated image of the original image by 
picture element Interpolating means 11 00 are convoluted by this enlarged edge infomnation so that a desired enlarged 
image is generated and outputted by Enlarged Images (El) output means 501 . In addition, there are interpolation 
methods such as the nearest neighbor method In which the value of the nearest sample is taken as interpolation value. 

30 [001 4] But the Interpolation methods mentioned above have some problems. In the linear interpolation method, the 
frequency characteristk:s in the pass band width is suppressed and the image data is smoothed as if the data is passed 
through a low pass filter (LPF), so that the image tends to blur with insuffk:ient sharpness and expression of details. 
The problem with the nearest neighbor method is that much lack of high frequency components occurs and that tends 
to cause distortion of the enlarged image, for example, jaggy in edge area and blunged mosaic distortion. 

35 [0015] The technique in (2) using the spatial frequency is proposed as a solution to that problem'. What those tech- 
niques aim at is to restore the high frequency component lost in data sampling and restore infonnatlon on details and 
edges of the image accurately, thereby improving the visual quality of the enlarged image. 

[001 6] FIG. 40 is a block diagram of a conventional system example to realize the technique in (2) that uses spatial 
frequency, while FIG. 41 schematkrally shows the processing procedure. First, an original image (n x n picture elements) 

40 in the real space as shown in FIG. 41 (a) is orthogonally transformed into an image (n x n picture elements) in the 
frequency space as shown in FIG. 41 (b). The image data in this frequency space is expressed in an n x n matrix, and 
the matrix obtained by this frequency transform shows lower frequency component as the position moves nearer the 
upper left of the figure and shows higher frequency component as the position moves in the right direction and in the 
downward direction along the arrows. 

45 [0017] Then in "0" component embedding means 4000, a s-fold area - an area of sn x sn shown in FIG. 41 (c) - of 
the transformed image in the frequency space is prepared. In the part of low frequency components in the sn x sn 
area, the frequency area of n x n shown in FIG. 41 (b) that is obtained by the orthogonal transfonm is copied, while the 
remaining part of high frequency component is interpolated with "0". Finally, in Inverse orthogonal transform means 
4001 , this frequency area of sn x sn is inverse-orthogonally transfonmed whereby a s-fold image data in the real space 

50 as shown in FIG. 41 (d) is obtained and an estimated enlarged image is outputted by Estimated Enlarged Image (EsEI) 
output means 4002 in FIG. 40. 

[0018] In addition to that method in whk:h the high frequency component is interpolated with "0", a technique is 
proposed in which the high frequency components are restored in a process of repeating transform and inverse trans- 
fomri of image data using orthogonal transform (method Invoh/ing Gerchberg-Papoulis repeat) as disclosed In unex- 
55 amined Japanese patent application 6-54172. Furthemiore, a rhethod as disclosed in unexamined Japanese patent 
application 8-294001 is proposed in which the orthogonal transfonn components in the original image are embedded 
in the low frequency area of the enlarged image, while the high frequency area is embedded with frequency Infonmation 
obtained on the basis of a prediction rule prepared in advance. 
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[0019] In the conventional image processing devices such as the digital camera, however, the image data once 
compressed and stored has to be expanded again, and in addition, the compression is perfomned by an lossy com- 
pression involving quantization in JPEG, and for this reason, the original image data not be reconstructed from the 
compressed image, and It often happens that some noise and cotor difference are caused. 
s [0020] Another problem is that color difference and Moire are caused by the thinning out at the time of resolution 
transfomi necessary for generation of a reduced image (hereinafter thumbnail image) to be displayed for review and 
therefore the smoothing by median filter or mean value fitter Is indispensable. But another problem is pointed out. That 
is, if the median filter or the mean value filter Is used, the sharpness of the thumbnail image will be lost and thesmoothing 
takes much time. 

10 [0021] Furthermore, the following problems are encountered with the conventional image enlargement required when 
a low-resolution image by the digital camera is printed or displayed. 

[0022] First, in the case of method (1) in which merely interpolation is perfonmed between the picture elements, it 
often happens that when an natural image is photographed by such a low-resolution device, the edge is not extracted 
with precision from the original image in the system shown in FIG. 42. As a result, the position where picture elements 
15 are interpolated and the interpolated data can not be adjusted well. To avoid this, an enlarged image is subjected to a 
repeat of processing by edge-enhancement filter. But the problem with the conventional edge-enhancement filter 
processing Is that unless a right filter is selected, the processing has to be repeated many times, or the edge is not 
enhanced at all. 

[0023] in method (2) in which the image is enlarged by embedding "0" in the high frequency area, on the other hand, 
20 a better enlarged image can be obtain than in the technique involving interpolating between the picture elements like 
the linear interpolation method and the nearest neighbor method. But since the high frequency components erased in 
data-sampling is not restored well, no sufficiently sharp image can be obtained. 

[0024] In the method In which the high frequency components are restored in the process of repeating right transf omn 
and inverse transfonm of an image by orthogonal transfomri, right transform and inverse transform are repeated, and 

25 that increases arithmetic processing. That is, the problem is the processing speed. The arithmetic processing amount 
Is not so troublesome if the enlargement ratio s is not so large, but if the enlargement ratio s is large, the arithmetk: 
processing amount of Inverse transform as opposed to the arithmetic processing amount of right transfomi Increases 
approximately in proportion to s x n. Especially in the two-dimensional processing actually perfomned, the arithmette 
processing amount increases roughly in proportion to the cube of s x n. Especially in enlargement of a color image, 

30 the enlargement for a plurality of color components will be necessary, further increasing the time needed for the process- 
ing. Furthemnore, in case the image to be enlarged is low in resolution, the high frequency components will not be 
restored to the accurately. 

[0025] The method disclosed In unexamined Japanese patent application 8-294001 takes those problems into con- 
sideration and considers the processing time and restoration of the high frequency components. This method involves 

35 embedding frequency infonnation obtained on the basis of a prediction rule prepared in advance in the high frequency 
area of the original image. Therefore, it Is necessary to work out a rule between the high frequency components and 
the other areas on the basis of a large number of picture samples in advance. It takes much labor to prepare a proper 
rule base, and If that can not be made into a proper rule, there is fear that It can not obtain suffci'ent effects. 
[0026] In addition, the image size is generally arbitrary, and the larger the size for orthogonal transfonn, the longer 

^ the processing is. Therefore, it is usual that the whole of an image of a specific size is not put to orthogonal transfomi 
at a time, but orthogonal transfomi is perfomned by blocks of a size of 4 pkrture elements to 1 6 picture elements. The 
problem is that it can happen that discontinuity between the blocks (block distortion) occurs In the border portion In an 
enlarged image. 

[0027] In view of those problems as set forth above, the present Invention has been made, and It is an object of the 
^ present invention to provide an. image processing device and image processing method whteh pennit reductk>n of 
memory size of image data and enlargement of an Image to a sharp and high-quality image. 
[0028] To achieve the object, the following means are adopted in the present invention. 

[0029] First, to reduce the memory size of an image data, as shown in FIG. 1 , an original image obtained from image 
Input means 1 0 Is orthogonally transformed by Original Images (Ol) orthogonal transfonning mean 11 so as to generate 

50 the frequency components of the original image, and the low frequency components are extracted from the frequency 
components of the original image by Low Frequency Components (LFC) extracting means 1 00. Then, High Frequency 
Components (IHFC) encoding means 13 works out the relation Information between the low frequency components 
and the remaining high frequency components In the frequency components of the original image and encodes the 
information, and at the same time Codes Synthesizing means 14 synthesizes the low frequency components and the 

55 relation infomnation into simplified image data. 

[0030] From the simplified image data thus generated. Low Frequency Components (LFC) decoding means 16 ex- 
tracts low frequency components, and at the same time, High Frequency Components (HFC) decoding means 1 7 takes 
out the relation informatton and decodes the high frequency components on the basis of the low frequency components. 
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Original Images (01) output means 18 combines the low frequency components and high frequency components and 
subjects the combination to inverse orthogonal transform to restore the original image. 

[0031] That pemnits the handling of simplified image data smaller in size than the original image and can produce a 

restored image close to the original image from a view of visual quality. 
5 [0032] The simplified Image data can be processed by inputting in means such as the personal computer which can 

restore an image, and also can be first stored in storage medium 15. Also, in the simplified image data, the data mount 

of the low frequency components can be further compressed by Low Frequency Components (LFC) compression 

means 300. In this case, it is desirable that a lossless compression method should be used. 

[0033] It is possible for Reduced Images (Rl) generating means 1 01 to extract the frequency area con-esponding to 
10 a specified size (preview size, for example) from the low frequency components and to generate a reduced image by 

perfomning Inverse orthogonal transfomnation on that component. 

[0034] In the present Invention, furthennore, when a specific image is enlarged to a desired size, Shortage Compo- 
nents (ShC) estimating means 500 estimates the shortage high frequency components on the basis of the frequency 
components of the image as shown in FIG. 5. And El output means 501 combines the frequency components of the 

15 specific image and the high frequency components obtained by ShC estimating means 500 and subjects the combi- 
nation to inverse orthogonal transfomri, thereby outputting an innage enlarged to a desired size. 
[0035] According to the present invention, furthenmore. In an image processing device for processing an original 
image inputted from image input means 10 as shown in FIG 7, Ol orthogonal transforming means 11 subjects the 
image data to orthogonal transf omi to generate the frequency component of the original image and from this frequency 

20 component of the original image. Enlarged Frequency (EF) estimating means 800 estimates the frequency component 
at the tinne when the original image is enlarged conresponding to some desired enlargement ratio. On the basis of the 
frequency component of the original Image thus obtained and the frequency component of the estimated enlarged 
image, EF estimating means 800 extracts the frequency component - as basic component - necessary for restoring 
the specified basic image to a predetemnined size, and multiple image encoding means 802 works out each relation 

25 information between the basic component and each frequency component corresponding to some estimated enlarged 
images and encodes the information. The basic component thus obtained and each relation information corresponding 
to some enlargement ratios are synthesized by Multiple Codes (MC) synthesizing means 803 to generate multiple 
simplified image data.. 

[0036] That makes It possible to generate multiple simplified image data corresponding to a plurality of sizes on the 
30 side where an image data is encoded. 

[0037] From the multiple simplified image data thus generated, the basic component and the relation infonnation are 
extracted, and the image data can be restored on the basis of the basic component and the relation information. 
[0038] In the multiple simplified Image data, too, the data size can be further reduced by the data size compression 
of the basic component. 

35 [0039] Secondly, to realize clear and high quality enlarged image data, inter-picture element interpolating means 
1100 performs interpolation between picture elements - according to a desired enlargement ratio - of image data in- 
putted from image input means 1 0 as shown in FIG 1 0. The interpolated enlarged Image thus obtained is not sharp In 
the edge portion, but with convolution means 11 01 performing a convolutional caknjiation to enhance the edge portion 
on the Interpolated enlarged image, an enlarged image with sharp edges is generated without tin^e-consumlng fre- 

40 quency transfonn. 

[0040] Thirdly, to realize a dear and high-quality enlargement of an image, Enlarged Frequency (EF) estimating 
means 120A estimates the frequency components of an enlarged image on the basis of the frequency conponents of 
the original image obtained by 01 orthogonal transfomiing means 11 as shown in FIG. 12. And on the frequency com- 
ponents generated by EF estimating means 120A, Inverse orthogonal transform means 1213 perfomns inverse orthog- 

45 onat transform corresponding to the enlargement size and obtains an enlarged image data. 

[0041] Estimation of the frequency components by EF estimating means 120A is performed using a linear approxi- 
mation or radial basis function network. The following method of estimating the high frequency components in an 
enlarged innage is also adopted. That is, from the original image, an edge image is taken out whtoh it is thought contains 
plenty of high frequency components, and from an enlarged edge image obtained by the linear transfomn, the high 

50 frequency components of an enlarged image are estimated using orthogonal transfonn. 

[0042] Furthemiore, the following method of estimating the frequency components of an enlarged edge image with 
high precision is used. That is, on the basis of the frequency components of the edge image taken out, the frequency 
components of the enlarged image is estimated with precision using the linear approximation or radial basis function 
network. 

55 [0043] Next, as shown in FIG. 23, block dividing means 2300 divides an inputted image data into a plurality of blocks 
taking into consideration the processing time required for the orthogonal transfonn. And Enlarged Block Images (Bl) 
frequency estimating means 2302 estimates the frequency components of the each enlarged image block for all of 
blocks divided from original image. Then, as shown in FIG. 24, the neighboring blocks are partiy overiapped, and on 
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the overlapped part in the enlarged block Images, the enlarged block generated later is adopted, thus reducing the 
' block artifacts. 

[0044] Furthemiore, the original image is processed by the transfonn function, thus reducing the discontinuity of an 
image on the block border-the discontinuity caused by division of an image into blocks. 
s [0045] In case the original image is a color image, Standard Component (SC) selec^ng means 2800 selects a color 
component to be a standard from among the color components making up an inputted color image and generates an 
enlarged image corresponding to the standard color components as shown In FIG. 28. 

[0046] And for the remaining color components, Shortage Component (ShC) enlarging means 2805 makes an esti- 
mation using the transfonn ratio derived by Transfonn Ratio (TR) deriving means 2801 from a color original image to 
10 the enlarged image of the standard color components, thereby speeding up the processing In generating an enlarged 
image data of a color image data. 

[0047] Fourthly, to realize a sharp and high-quality enlargement of an image, also, multiple resolution analysis in 
Wavelet transform is utilized. 

[0048] For this, when an original image with n picture elements x m pk;ture elements is enlarged to make an enlarged 
'5 image with Ln picture elements x Lm picture elements, Input Images (II) regulating means 12 interpolates or thins out 
(hereinafter both expressed as "regulate") to Ln/2 picture elements x Lnn/2 pkiture elements as shown in FIG. 29. And 
to the Image regulated by II regulating means 12, image enlarging means 290A applies a method based on Wavelet 
transfonn and generates an enlarged image. 

[0049] In the enlargement, to be concrete, the images - an image regulated according to the number of picture 
20 elements of an enlarged Image to be obtained, the edge image In the vertical direction , the edge image in the horizontal 
direction and the edge image in the oblique dire(^k>n - are regarded as four sub-band images making up a Wavelet 
transfonn Image. And by perfonning Inverse Wavelet transfonn on the suk^tend images, an enlarged Image of a desired 
picture element size is to be obtained. 

[0050] Also, in the enlargement, the relation Is found between three edge images obtained from a 1/4 size reduced 
25 image in the low frequency area of transfonn image data, that is, Wavelet transfonned original image data and the 
remaining three sub-band images within the transfonn Image data. Here, the edge image in the vertk»l direction, the 
edge image in the horizontal direction, the edge image In the oblique directton of the image regulated according to the 
number of pk:ture elements of an enlarged image to be obtained are each corrected using the above relation infomna- 
tion. 

30 [0051 ] And the regulated image and the three corrected edge images are regarded as four sub-band images making 
up the transfonn image data. By performing inverse Wavelet transform on the sub-band images, an enlarged image 
of a desired picture element size is obtained. 

[0052] In the enlargement, furthermore, the relation Is to be found between one typical edge image data obtained 
from the 1/4 size reduced Image data present in the low frequency area of the transform image data and the remaining 
35 three sub-band image data within the transform image data. Here, one typical edge image obtained from the image 
• regulated according to the number of pk:ture elements of an enlarged image data to be obtained is conrected using 
the relation information. 

[0053] And the regulated image data and the three image data obtained by correcting are regarded as four sub-band 
images making up the transfonn Image data, and Inverse Wavelet transform Is perfonned to obtain an enlarged Image 

40 of a desired picture element size. 

[0054] Also, as shown in FIG. 37, Enlarging Process (EP) initializing means 3700 sets an original image as object 
image for enlargement, and Object Images (Obi) enlarging means 3701 applies a method based on Wavelet transfonn 
to the object Image for enlargement to generate an enlarged Image having four times as many picture elements. And 
Multiple Processing (MP) ending judge means 3703 sets the enlarged image obtained by Obi enlarging means 3701 

^ as object image for enlargement and retums the process to Obi enlarging means 3701 . Enlarged Images (El) presenting 
means 3702 presents visually the enlarged image obtained from Obi enlarging means 3701 . Furthermore, image fine- 
adjustment means 3704 enlarges or reduces the enlarged image presented by El presenting means 3702. 
[0055] To be concrete, in case the image size to be enlarged is not known now, the above enlargement system using 
the Wavelet transfonn is adopted. With the enlarged image obtained by inverse Wavelet transform as next object 

50 image, inverse Wavelet transfonn is performed. That is, enlargement to an image having four times as many picture 
elements is repeated. And when the enlargement reaches a visually desired enlargement ratio, the process is sus- 
pended. That way, enlargement of the inputted original Image is performed. 

[0056] In case the original image is a color image. In the enlargement method using the Wavelet transfonn, too, a 
color component as standard is selected from among the color components making up the color image, and for its 
55 standard color components, an enlarged image is generated. 

And the remaining color components are found by perfonning linear transfonn on the enlarged image of the standard 
color component using a ratio of each remaining cobr to standard component, thus speeding up the processing in 
generating an enlarged Image of a color image. 
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[0057] Having summarized the invention, a detailed description of the invention foiiows with reference being made 
to the accompanying drawings which form part of the specification, of which: 

FIG. 1 is a block diagram showing the arrangements of first, second and third embodiments of the present invention. 
5 FIG. 2 is a schematic diagram showing the processing steps of the first embodiment of the present invention. 

FIG. 3 is a block diagram of the fourth enrtbodiment of the present invention including an image processing device. 
FIG. 4 Is a schemata diagram showing the procedure for spatial predk:tlon and compression used In the four 
embodiment of the present invention. 

FIG. 5 is a block diagram of the fifth embodiment of the present invention including an image processing device. 
10 FIG. 6 is a schematic diagram showing the procedure of means for estimating shortage components used.in the 

fifth embodiment of the present Invention. 

FIG. 7 is a block diagram showing the arrangements of sixth, seventh and eighth embodiments of tiie present 
Invention. 

FIG. 8 is a schematic diagram showing the processing steps of means for extracting bask; components and multiple 
IS image encoding means used in the sixth embodiment of the present invention. 

FIG. 9 is a block diagram of the ninth embodiment of the present invention including an image processing device. 
FIG. 10 is a block diagram showing the arrangement of a tenth embodiment of the present Inventton. 
FIG. 11 is a schematk; diagram showing the procedure of the innage processing device of the tenth embodiment 
of the present invention. 

20 FIG. 12 is a block diagram showing the anBngement of an eleventh embodiment of the present invention. 

FIG. 13 is a diagram schematically showing the processing steps of an image processing device of the eleventh 
embodiment of the present invention. 

FIG. 1 4 is a schematic diagram of the radial basis function network. 

FIG. 15 is a diagram schematically showing the relation between an original image and an enlarged image in the 
25 Image processing device of the eleventh embodiment of the present Invention. 

FIG. 16 is a block diagram showing the arrangements of an image processing device of the twelfth embodiment 
of the present invention. 

FIG. 17 is a diagram schematically showing the processing steps of the Image processing devk» of the twelfth 
embodiment of the present invention . 
30 FIG. 1 8 is a diagram showing Laplacian filter examples, 

FIG. 1 9 Is a block diagram showing the an^angements of the Image processing device of the thirteenth embodiment 
of the present invention. 

FIG. 20 Is a diagram schematteally showing the processing steps of tiie Image processing devtee of the thirteentti 
embodiment of the present Invention. 
35 FIG. 21 is a diagram schematically showing the processing in case linear approximation Is used by means for 

generating edge frequency of the thirteenth embodiment of the present invention. 
FIG. 22 Is a diagram schematk:ally showing the distortion between the enlarged blocks. 
FIG. 23 Is a block diagram showing tiie arrangement of the image processing devk:e of a fourteenth embodiment 
of the present invention. 

40 FIG. 24 is a diagram schematically showing the processing steps of the image processing device of the fourteenth 

embodiment of the present invention. 

FIG. 25 is a block diagram showing the arrangement of the image processing devk:e of a fifteenth embodiment of 
the present Invention. 

FIG. 26 Is a diagram schematically showing the processing steps of the Image processing devbe of the fifteenth 
45 embodiment of the present invention. 

FIG. 27 is a diagram schematteally showing the processing steps by means for transfomning data within the block 
of the image processing devrce of the fifteenth embodiment of the present invention. 

FIG. 28 is a block diagram showing the arrangements of ttie image processing devtee of a sixteenth embodiment 
of the present invention. 

so FIG. 29 is a block diagram showing the anBngement of the Image processing devtee of a seventeenth embodiment 

of the present invention. 

FIG. 30 is a diagram showing the filter processing steps in Wavelet transform coefficient. 
FIG. 31 is a diagram schematcally explaining the sub-band components in a Wavelet transfomr) coefficients. 
FIG. 32 is a diagram schematically showing the estimation of shortage sub-band component in a transfomn image. 
ss FIG. 33 shows filter examples applied for detectbn of edges in the vertcal direction, horizontal direction and oblique 

direction. 

FIG. 34 is a block diagram showing the arrangements of enlarging images means making up the image processing 
device of an eighteenth embodiment of the present invention. 
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FIG. 35 is a diagram schematically showing the estimation and correction of shortage sub-band components in a 
transfomi image. 

FIG. 36 Is a block diagram showing the an-angement of means for enlarging Images maidng up an image processing 
device of a nineteenth embodiment of the present invention. 
5 FIG. 37 is a block diagram showing the an^angement of the Image processing device of an twentieth embodiment 

of the present invention. 

FIG. 38 Is a block diagram showing the arrangement of the image processing device of a twenty-first embodiment 
of the present Invention. 

FIG. 39 is a block diagram showing the anangement of a conventional digital camera. 
10 FIG. 40 is a block diagram showing the arrangemerit of a conventional Image enlarging devk» In which an image 

Is transfomied in the frequency area and enlarged. 

FIG. 41 is an explanatory diagram showing an example in which a conventional Image Is transfomied In the fre- 
quency area and enlarged. 

FIG. 42 Is a block diagram showing the arrangement of a conventional image enlarging devbe in which an image 
15 is enlarged Interpolation between picture elements. 

FIG. 43 is a schematic diagram showing the conventional interpolation between picture elements. 

[0058] Now, the embodiments of the present Invention will be described with reference to the drawings. 
[0059] It Is to be understood that hereinafter the unit of coordinate values is all identical with the unit of the distance 
20 between picture elements. Also, hereinafter, the original image that will be described by way of example is an original 
Image taken in by scanner, digital camera or the like. But It Is not restrictive. The original Image may be a specific Image 
or the like held on a magnetb disk etc. or an image or the like sent in through communication means like the Intemet. 

Embodiment 1 

[0060] First, there will be described the image processing device of Embodiment 1 of the present invention. 
[0061] In FIG. 1 , an original Image of a size N picture elements x N pteture elements obtained by image input means 
10 like CCD elements Is transformed Into component data (frequency component data) In frequency space. For that, 
orthogonal transfonm is used, and among the transform method are Hadamard transfomn and fast Fourier transform 
30 (FFT), discrete cosine transfomri (DCT), slant transform, and Haar transfomri. Here in this description, DCT is used. 
But the present Invention is not limited to that transfonn technique but other orthogonal transfomr) methods may be 
applk»ble. 

[0062] It is also noted that DCT used here Is two-dimensional DCT especially for dealing with images, and the trans- 
fomi fomiula Is given by Formula 2. 

35 

Fiu.v) « 7jc{u)c(y)/^K_xK_y 
40 x^' g [i)(jr,y)cos((2;t+l>i;r/2A:^jc)cos((2^ 

c(0)-l/V2 

where F (u, v) represents a DCT component at the component position (u, v), and D (x, y) represents Image data at 
pk:ture element posltton (x, y). Furthenmore, K _x indicates the number of picture elements In the x direction and K _y 
50 indk^ates the number of picture elements In the y direction. But here in this embodiment, a square Image of N k N is 
take up, that is, K _x = K _y = N. 

[0063] LFC extracting means 1 00 extracts low frequency components of the frequency component data of the original 
Image as shown in FIG. 2 (b). 

[0064] H FC encoding means 1 3 encodes the high frequency area (e) that are left when the low frequency components 
55 (c) are taken out from the frequency components of the original image in FIG. 2 (b). Here in this embodiment, the area 
of the high frequency components is divided into snnall blocks HI to H5 as shown In FIG. 2 (e), for example. Likewise, 
the area of lowf requency component (c) is divided Into blocks L1 to L4as shown in FIG. 2 (f). And frequency components 
within the respective blocks HI to H5 of the high frequency component (e) and the respective blocks LI to L4 of low 
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frequency components (0 are made related to each other. For example, It may be in the fomi of linear expression 
multiplied by certain constant coefficients as In Fonnula 3, or the frequency components within the respective blocks 
H1 to H5 may be approximated by mutti-dimensional function v (LI , L2, L3. L4) with the frequency components of the 
respective blocks LI to L4 as variable. 

5 

Fomnula 3 
[0065] 

10 

H1=aL2 
H2 = L4 

H4=L3 
H5 = y'L2 

20 

[0066] Within the respective blocks, furthermore, it is not necessary to have the same coeffbient. Instead, for re- 
spective blocks HI to H5, coeffblent matrix M _C1 to M _C5 are provided, and using them and low frequency component 
matrix M _L1 to M _L4 made lip of Ifrequency components within blocks L1 to L4, it is also possible to express high 
frequency component matrixes M_H1 to M _H5 made up of frequency data within blocks H1 to H5. In any of the 
25 methods, the high frequency data size can be reduced by formulating, the remaining high frequency components In 
FIG. 2 (e) expressing the deamess of the image and edge infomnatton using the low frequency components data in 
FIG. 2 (c). 

[0067] Codes Synthesizing means 14 synthesizes the code string - of the relation infonmatton between the low fre- 
quency components and high frequency components obtained from a rule description obtained at HFC encoding means 

30 1 3 or data such as coefficients of the approximation expression used by HFC encoding means 13 - and low frequency 
components data of the original image obtained from LFC extracting means 100. And the Codes Synthesizing means 
14 stores the data on the storage medium 15 as simplified Image data. Then, to read and restore data, a table is 
provided within the storage medium 15, and Infomnation-the number of stored Image data, the sizes of the respective 
image data, the ratb of low frequency component data in the respective data stored, the ratio of codes indicating high 

35 frequency components - is stored, thus making extraction of the respective data efficient. 

Embodinrtent 2 

[0068] Next, there will be described the image processing device of Embodiment 2 of the present invention. 
40 [0069] In FIG. 1, the processing steps by image input means 10 and 01 orthogonal transfomiing means 11 for or- 
thogonal transfonn of original images are the same as those in Embodiment 1 and will not be described. LFC extrac^ng 
means 1 00 extracts low frequency components according to the number of picture elements of Reduced Images (Rl) 
display means 102 that displays thumbnail images. Rl generating means 101 performs inverse orthogonal transform 
on the low frequency components, and thumbnail images are displayed on Rl display means 102. 

45 

Embodiment 3 

[0070] Next, there will be described the Image processing device of Embodiment 3. 

[0071 ] In FIG. 1 , the user gh^s an Instruction signal by inputting means such as a keyboard, mouse etc. (not shown) 
so so as to take out desired data from simplified image data stored on the storage medium 15 by the image processing 
device in Embodiment 1 of the present invention and output the data on a high resolution laser printer or ink jet printer 
or to edit the data on CRT. According to this instruction signal, LFC decoding means 16 first takes out low frequency 
components as the main data from the storage medium 1 5. And HFC decoding means 1 7 decodes the low frequency 
components obtained by LFC decoding means 1 6 and the high frequency components according to relation Infomnation 
55 between the low frequency components and high frequency components within the storage medium 15. Then, Ol 
output means 18 combines the low frequency component data and high frequency component data and perfomns 
Inverse orthogonal transfonn of the corresponding image size so as to output data to be handled t)y other Image 
processing devk:es. that is, to be displayed on CRT, or to be outputted on the printer or the tike. 
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[0072] Because image data stored on the storage medium 15 are not quantized when the data is encoded, the 
original image thus obtained by this method is sharper than that by the prior art. 

Embodiment 4 

5 

[0073] Next, there will be described the image processing device of Embodiment 4 of the present invention. 
[0074] In FIG. 3, the processing steps by Ol orthogonal transforming means 11 , means 1 00, HFC encoding means 
13, Codes Synthesizing means 14 and the storage medium 15 is the same as in the image processing device of 
Embodiment 1 , and will not be explained. Considering that the volume of the low frequency component data of the 
10 original image obtained by LFC extracting means 100 is large in size and in consideration of the restriction by the 
volume size of the recorded medium etc., the low frequency component data is compressed by LFC compression 
means 300. 

[0075] The processing method by LFC compression means 300 is based on differential encoding (Differential PCM; 
DPCM) and entropy encoding called the spatial method in JPEG instead of compression technique using usual OCT 
IS and quantization, entropy encoding (the compression technique also called the baseline technique). It is desirable to 
use the lossless compression method that will not cause distortion by compression and expansion. 
[0076] In this technique, prediction values are worked out by a predictor using the neighboring density values, and 
from the density value to be encoded is subtracted its prediction value. For this predictor, seven kinds of relational 
expressions are made available as shown in Formula 4. 



20 



25 



35 



Fomnula 4 
[0077] 



dDx = D1 

dDx^DZ 

dDx=D3 
dDx=D34^(D2' D1) 
d[}x = D3 + (D2'Diy2 
dDx==D2 + (P3-Diy2 
dDx (D2 + D3y2 



40 [0078] With the density value to be encoded as Dx, the neighboring three density values are to be called D1 , D2, 
D3 as shown in FIG. 4. TTie prediction value dDx to be cabulated from those three density values is defined by Fonmula 
4. Whteh expression is used is written in header infomiatlon of the compressed image data. In encoding, one predbtion 
expression is selected as shown in FIG. 4 (b), and subtracted Ex is worked out. This subtracted Ex is entropy encoded. 
Encoding prediction errors makes rt possible to compress the low frequency components reversibly. 

45 [0079] On the basis of the low frequency component data compressed by LFC compression means 300, HFC en- 
coding means 13 fonnulates the remaining high frequency components in a way as shown in Formula 3 as in Embod- 
iment 1 . 

[0080] Codes Synthesizing means 1 4 synthesizes the compressed frequency component data and the code string 
of the relation infonnation between the low frequency components and high frequency components obtained from the 

50 rule description obtained by HFC encoding means 1 3 or data such as the coefficient etc. of the approximation expres- 
sion used by HFC encoding means 13, and stores the data on the storage medium 15 as simplified image data. 
[0081] According to this embodiment, it is possible to compress the low frequency components to 1/2 to 1/4 whk:h 
It is thought accounts for the largest portion in size in the data size of the devk^e in the Embodiment 1. Also, by using 
a lossless compressk>n method for compression, it Is possible to store data without impairing the frequency components 

55 possessed by the original image. 

[0082] It is noted that since the low frequency component data is a basic image to restore high frequency components 
whrch express the sharpness in the details of the image, a lossless compression method is applied to the low frequency 
components of the original image to realize the outputting of a clearest image with a sharp edge. But the usual JPEG 
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baseline method can also be used here. It Is a lossy compression, and when it is applied the cleamess of the image 
will be lost. But the data size can be compressed to 1/1 0 to 1/20, and rt is a method that is used as in case the image 
volume to be held on memory media and to be read is compacted as far as possible so that much image data can be 
stored. 

5 [0083] Also, the relation infomnation between the low frequency components and high frequency components ob- 
tained from HFC encoding means 13 in the image processing device of Embodiment 1 and the relation infonnation 
between the basic components and the high frequency components of the respective enlarged images obtained from 
multiple image encoding means 802 in the image processing devices in Embodiments 6 and 9 can be compressed by 
lossless compression methods such as the spatial method. 

10 

Embodiment 5 

[0084] Next, there will be explained the image processing device of Embodiment 5. 

[0085] In FIG. 5, the processing steps by storage medium 15, LFC decoding means 16 and HFC decoding means 

IS 1 7 is the same as in Embodiment 3 and will not be explained. 

[0086] ShC estimating means 500 is provided with a function of estimating the high frequency component data that 
will be short In enlarging the data to a desired size In case the original image read Is displayed on high-resolution CRT 
etc. El output means 501 combines the shortage of high frequency components estimated by ShC estimating means 
500 and the frequency components of the original image decoded from the encoded image data stored on the storage 

20 medium 15 by LFC decoding means 16 and HFC decoding means 17, and perfomris inverse orthogonal transfomi 
corresponding to the enlargement size, thereby generating an enlarged image of the original image read. 
[0087] ShC estimating means 500 and El output means 501 process data as shown in FIG. 6. First, the frequency 
component data as shown in FIG. 6 (b) obtained by perfonning orthogonal transfonn of an original image of a size N 
picture elements and N picture elements as shown in FIG. 6 (a) are embedded in the low frequency area of the frequency 

25 components of an enlarged image having a coefficient size conresponding to a desired enlargement size (sN picture 
elements x sN picture elements)(FIG. 6 (c)). And the shortage components H1 to H3 that occur then are estimated 
from the frequency components of the original image shown in FIG. 6 (b). There are a number of methods of estimation 
including: 

30 (1) Method in which nonlinear approximation capacity of the radial basis function networi( (RBFN) is applied. 

(2) Method in which from the frequency components of the edge image of the original image, the frequency com- 
ponents of its enlarged edge image are estimated with precision using linear approximation or radial basis function 
network, whereby the high frequency components of the enlarged image which were lacking from the original 
image are estimated. 

35 (3) Method in whk:h as disdosed in unexamined Japanese patent application 8-294001 , the orthogonal transfomi 

components of the original image are embedded in the low frequency area and the frequency information obtained 
on the basis of a prediction mie prepared in advance are embedded in the high frequency area. 
(4) Method in whbh the high frequency connponents are restored in a process of repeating transform and inverse 
transfonn of image datas using orthogonal transfonm as proposed in unexamined Japanese patent application 

40 6-541 72 (method based on Gerchberg-Papoulis repeating). 

(Methods (1) and (2) will be described later.) 

[0088] In addition, a number of techniques of enlarging Images using the frequency area are available. But the ShC 
^ estimating means 500 requires a technique of estimating with precision of the high frequency components necessary 
to generate a clear image in details with sharp edge. If this requirement is satisfied, any technique is applicable. 
[0089] And El output means 501 perfomris inverse orthogonal transfomi con-esponding to the coefficient size of sN 
x sN as shown in FIG. 6 (d), whereby the frequency coniponents of the enlarged image estimated by ShC estimating 
means 500 are brought back to real space data, and outputted as data to be handled by other image processing devk^es 
so as to be displayed on CRT or the like, or to be referred to an outputting unit such as the printer 

[0090] Through such an-angements, the high frequency component data required in outputting an enlarged image 
using frequency components of the original image are estimated and compensated, thus avoiding a blurred edge and 
unclear details which are observed in enlargement by the prior art pteture element interpolation. 

ss Embodiment 6 

[0091 ] Next, there will be explained the image processing device of Embodiment 6 with reference to FIG. 7 and FIG. 8. 
[0092] In FIG. 7, the processing by Image input means 1 0 and Ol orthogonal transforming means 11 is identical with 
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that in Embodiment 1 of the present invention and will not be explained. First, using the frequency components of the 
original image obtained by Ol orthogonal transfomning means 11 , EF estimating means 800 estimates the frequency 
component data in enlarging the image to a plurality of image sizes. The estimating method can be the same technique 
as ShC estimating means 500 of FIG. 5, one of the constituent elements of the image processing device in Embodiment 

5 5 of the present invention. 

[0093] Then, the basic components are extracted by Basic Components (BC) extracting means 807. This process 
Is schematically shown in FIG. 8. FIG. 8 (a) shows the frequency components of the original image, FIG. 8 (b) shows 
the frequency components of an image produced by enlarging the original image twice vertically and twice horizontally, 
and FIG. 6 (c) shows the frequency components of an image produced by enlarging the original image three times 

10 vertically and three times horizontally. More image sizes may be provided, and the enlargement ratio of the image size 
does not have to be an Integer. In FIG. 8 (a), (b) and (c), the frequency components in the low frequency area represent 
the whole-wise feature of an image and have a qualitative tendency, which is common to the respective enlargement 
ratios. Therefore, frequency component LOO in a low frequency area that exhibits a similar tendency is to be taken as 
a basic component as shown in FIG. 8 (d). For simplification of processing, the frequency components of the original 

15 image shown in FIG. 8 (a) may be regarded as basic component. 

[0094] Multiple encoding mearts 802 in FIG. 7 relates the basic component LOO and the respective blocks H11 to 
HIS shown in FIG. 8 (b) and fomnulates them (FIG. 8 (e-1)) In the same way as HFC encoding means 13 does in the 
image processing devices of Embodiments 1 and 4 of the present invention. The respective blocks H21 to H25 shown 
in FIG. 8 (c) also can be related directly to the basb component LOO in the same way. Here in the present embodiment, 

^ however, by relating the respective blocks H11 to H13 in FIG. 8 (b) and the respective blocks H21 to H25 in FIG. (c) 
as shown in FIG. 8 (e-2), the basic connponent (frequency component in a low frequency area) LOO and the respective 
blocks H21 to H25 In FIG. 8 (c) are indirect related and fonnulated. That Is because less qualitative difference is 
observed between the near blocks in the frequency area, the accuracy by the relating procedure. 
[0095] MC synthesizing means 803 synthesizes the extracted base component and the relation information between 

25 the bask: component obtained by multiple Image encoding means 802 and the respective frequency components of 
the enlarged images and stores data on the storage medium 15 as multiple simplified image data in the same way as 
Codes Synthesizing means 14 does in Embodiments 1 and 4. Then, it is necessary to write the number of enlarged 
images prepared multiple-wise, the starting flag signals of data of the respective sizes etc. on the headers or the like 
of the multiple simplified image data. 

30 [0096] As set forth above, if enlarged frequency components corresponding to a plurality of image sizes, matched 
to the resolution the outputting device, are prepared in advance, it is possible to save the time for estimating the high 
frequency components that get short in outputting an image of a desired size on the basis of data read from the memory 
medium, thus improving the user interface. 

35 Embodiment 7 

[0097] Next, there will be explained the image processing device of Embodiment 7. 

[0098] In FIG. 7, the processing steps by image input means 1 0, 01 orthogonal transfonning means 1 1 , EF estimating 
means 800 and BC extracting means 807 are the same as those in the Image processing device of Embodiment 6 of 

^ the present invention and will not be explained. 

[0099] Basic Image (Bl) generating means 808 perform inverse orthogonal transform on the baste component ex- 
tracted by BC extracting means 807. In case the size (coeffkHent size) in the frequency space of the frequency com- 
ponents extracted by BC extracting rheans 807 is larger than the size of the image for review on the display, a frequency 
component matched to the resolution will be further extracted from the low frequency area of the basic component and 

^ a bask: image for display is generated. If, on the other hand, the coeffteient size of the frequency components extracted 
by BC extracting means 807 is smaller than the size of the image for review on the display, '0" component shown In 
FIG. 41 will be embedded In the area where the coefftelents of the basic component are short, and a thumbnail image 
is generated. But it is possible to include the following processing in Bl generating means 808. That is, in such a case, 
the shortage components Is estimated according to the image size as EF estimating means 800 and the Image is 

50 enlarged to the image size for review. 

Embodiment 8 

[0100] Next, there will be explained the image processing device of Embodiment 8. 
55 [0101] In FIG. 7, the user can select an image, an object for outputting and editing, from among the image data 
stored on the storage medium 15 by the image processing device of Embodiment 6 of the present invention and also 
can specify the image size. In this case, the following methods can be used. That is, selection Infomnation according 
to the image sizes stored on the storage medium 15 is presented, or the user inputs an enlargement ratio with the 
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original image size as basis, or the size is automatically selected in accordance with the resolution of equipment to 
which the image is outputted. 

[0102] First, there will be described the processing in case selection infomnation according to the image sizes stored 
on the storage medium 1 5 is presented. The user selects and specifies a desired image and desired image size by 

3 inputting means such as the mouse (not shown). From the storage medium 15, Basic Components (BC) decoding 
means 804 takes out the basic component of the image from the multiple frequency components corresponding to the 
Instruction. Then, Object Frequency (ObF) decoding means 805 takes out the expression code string of the high fre- 
quency component data corresponding to the selected image size, and decodes the high frequency components other 
than the basic component of the object enlarged image size from the code string and the basic component. Object 

10 Images (Obi) output means 806 combines the high frequency components and the basic component, perfomns inverse 
orthogonal transform and outputs the desired enlarged image. 

[0103] Next, there will be explained the processing in case the enlargement ratio with the original Image size as 
basis is inputted or the image size is automatically selected in accordance with the resolution of equipment for output- 
ting. In case image sizes are not kept multitudinously on the storage medium 15, the image data of the size nearest 

IS the enlarged size is selected, and processed by BC decoding means 804, ObF decoding means 805, and then Obi 
output means 806. Then, it is to be understood, in Inverse orthogonal transform in Obt output means 806, inverse 
orthogonal transfomi coniesponding to the image size nearest the size after enlargement is perfonned. 
[01 04] At this stage, however, it Is also possible to select an image size smaller than the desired image size and the 
shortage is made up by ShC estimating means 500 in Embodiment 5. 

20 [0105] As set forth above, if the low frequency component that can be regarded as common to the frequency com- 
ponents of a plurality of enlarged sizes is extracted as basic component and the remaining high frequency components 
of the respective enlarged sizes are encoded on the basis of this bask: component, frequency components of image 
on a plurality of enlarged sizes can be provided. Therefore, when an enlarged image of the needed image size is 
reproduced according to the user's instruction, the enlarged image can be reproduced at a high speed without esti- 

25 mating the shortage of high frequency components in the desired enlarged size each time an Instru^on is given. 

Embodiment 9 

[0106] Next, there will be explained the image processing device of Embodiment 9. 
30 [01 07] In FIG. 9, the processing steps by image input means 1 0, Ol orthogonal transforming means 11 , EF estimating 
means 800, and BC extracting means 807 is the same as those in the image processing device of Embodiment 6 of 
the present invention, and will not be explained. 

[0108] Basic Components (BC) compression means 1000 compresses the basic component extracted by BC ex- 
tracting means 807. This compression step is identical with that by LFC compression means 300 in the image process- 
35 ing devk:e of Embodiment 4 of the present invention and will not be explained. 

[0109] Also, the processing by multiple image encoding means 802, MC synthesizing means 803 and the storage 
medium 15 following that by BC compression means 1000 is the same as that In the image processing devk» of 
Embodiment 6 of the present invention and will not be explained. 

[01 10] Through such arrangements, it is possible to reduce image data size - the problem that arises when a plurality 
40 of image sizes are held. 

Embodiment 1 0 

[01 1 1 ] Next, there will be explained the image processing device of Embodiment 1 0. 
45 [01.12] FIG. 10 is a block diagram showing the anrangement of the device. 

[0113] First, interpolating is effected by an interpolating technique shown in FIG. 43 between the picture elements 
on an original image read by image input means 10, and inter-picture element interpolating means 11 00 generates an 
enlarged image. Then, convolution means 1101 repeats convolution of the enlarged image for enhancing the picture 
elements. 

50 [01 14] FIG. 1 1 shows the convolution process. In case the edge is an unclear image as shown in FIG. 1 1 (b), Image 
data on the edge can not be extracted, and it is difficult to enlarge the image in the processing step as shown in FIG. 
42. But according to the present invention, the averaged density value can be enhanced without difficulty. It is noted 
that the processing by convolution means 1 1 01 produces the same effect on the enlarged image as when the processing 
t>y the edge enhancing filter is repeated several times. The conventional processing by edge enhancing filter has 

55 problems that the processing has to be repeated many times, or the edge is not enhanced at all unless a suitable filter 
is selected, but the convolution method has no such problem because in convolution means 1101, convolution is 
effected by the density value itself. If it is assumed that when convolution is performed at point P in FIG. 11 (c) for the 
K-th time with that density value as Dp [K], the density value Dp [K + 1 ] at point P by convolution for the (K + 1 )4h time 
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is defined as in Fof an objeconnula 5. This is to find the average value of convolution values near that picture element. 
Here, £1 represents the range of an object picture element for convolution, U _ Gaso represents the total number of 
picture elements within its Q, and q represents any picture element within Q. 

5 Formula 5 



[0115] 



Dp[k + 1] - (S(Dp[k] X Dq[k])yU_Gaso 



[01 16] If it is assumed that the set of picture elements contained in the enlarged interpolated Image Is the whole of 

15 the star A, all the picture elements within this A will be processed. 

[0117] Next, convergence judging means 1102 judges the convergence of this convolution processing. It judges 
whether on the basis of the mean value within A of the square enters between the density value Dp[lc - 1] obtained at 
the(K- 1)th convolution and the density value Dp[K]atthe K>th convolution as in Fomnula 6 is satisfied with convergence 
judgment value. It judges that the convergence of the density value is over and finishes the convolution if the mean 

20 value is smaller than a preset convergence judgement value Thred, and estimated enlarged image outputting means 
1312 outputs the image as estimated enlarged image. 

Fonmula 6 



25 [0118] 



Thred ^ {^(Dp[k]-Dp[k -l]f)/T_Gaso 

30 

In Fomiula 6, T_Gaso represents the total number of picture elements within A. 

[0119] As set forth above, the Image processing device of present embodiment is to enhance an enlarged image 
through convolutional arithmetic processing of interpolated image data and can realize an enlarged image in a handy 
35 manner without conducting time-consuming frequency transform. Also, it is possible to obtain an enlarged image with 
edge information without difficulty in processing even an original image with unclear-cut edge which presents a problem 
when edge infomiatlon of the original image is used to keep an interpolated Image from blurring. 

Embodiment 11 

40 

[0120] FIG. 12 shows the arrangement of the image processing device of Embodiment 11 . 
[0121] Using original image data of n picture elements x n picture elements inputted in this device by Image input 
means 10, Approximate Coefficient (AC) deriving means 1200 within EF estimating means 120A derives - in a method 
which will be described later - approximation coefficient vector (weight coefficient vector) Vw = (w _0,w _1 w _N-1) 
^ (N = n X n) which is used in the rionlinear interpolation method applied in estimating the frequency component data 
of the enlarged image. 

[01 22] Using weight coefficient vector Vw obtained by AC deriving means 1 200 and the frequency component data 
of the original image obtained by Ol orthogonal transfonning means 11 , non-linear estimating rheans 1201 in EF esti- 
mating means 120A estimates the frequency component data of an enlarged image - as will be described later - using 
50 the radial basis function network. 

[0123] Now, there will be explained the operation of this image processing device. 

[0124] Image input means 1 0 reads out data of an original image of a size of n pk^ture elements x n pk;ture elements 
for enlargement. The original image obtained by image input means 1 0 is processed in the same way as in Embodiment 
1 until the original image is transformed into component data (frequency component data) in frequency area by Ol 
55 orthogonal transforming means 11 , and the processing until that will not be described. 

[0125] On the basis of density value D{x, y)(x, y =0, n-1 ){FIG. 13 (a)) of the original image from image input means 
10 and frequency component data F(u, v)(u, v=0.,... n-1) (FIG. 13 (b)) from Ol orthogonal transfonning means 11 , EF 
estimating means 1 20A estimates frequency component data F'(u, v)(u,v=0,. .., m-1 ) (FIG. 1 3 (c)) of an enlarged image 
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at the time when the original innage of n x n is enlarged to an m x m Image (n < m = s x n). This estimation is made 
using the radial basis function network (hereinafter RBFN) which will be described later. 

[0126] Not a linear but an non-linear relation is generally applicable between the two-dimensional DCT component 
value and its component position (u, v). For this reason, a technique that can approximate this non-linear relation with 
s precision is required to produce a clear enlarged image without the edge getting blurred. Among such techniques are 
one using hierarchical neural networic leamed on the basis of learning data prepared in advance and a technique of 
fuzzy logic reasoning using a rule base extracted from leaming data prepared in advance. For these techniques, how- 
ever, it is indispensable to prepare much image data for learning in advance so that estimation can be made with high 
precision. 

10 [01 27] On the other hand, there are techniques that can self-organize approximation function using the features of 
inputted data without leaming data unlike the above-mentioned techniques. One example of these is a technique that 
makes short-term prediction of time series pattem that fluctuates chaotically, and the radial basis function network that 
is used here is considered to be a kind of that technique. Here, the surface drawn by frequency components is approx- 
imated by non-linear function using RBFN from the frequency components of a given image and its corresponding 

15 position. 

[0128] FIG. 14 is a example model anrangement of RBFN used here. To this, the component position vector VP = 
(u, V) to be found is given as Input, and unified into one at the outputting layer through transf orni by radial basis function 
(RBF) ^ at the intermediate layer, and output frequency component Fout = F (u, v) is outputted. This is shown in 
Formula 7 where VPJ = (uj, vj) represents the center of the i-th RBF function and N represents the number of RBF 
20 functions. 

Formula 7 

[0129] 

25 




30 {Ossu v^m -1} 

[0130] If, for example, an original image Is enlarged with If requency components P as center as shown in FIG. 15, 
VPJ corresponds to the position vector of the frequency component of the original image, and the number N of the 

35 RBFfunctions con'esponds to the number n x n of picture elements of the original image. That is, as many RBF functions 
as the number of pbture elements of the original image are provided, and frequency component data after enlargement 
placed around component position vector VPJ at P, the center of the RBF function, Is estimated as overiapping of 
these RBF function outputs. It is noted that RBF is a function that changes depending on distance |{ VP • VPJ || 
between component position vector VP and the center VPJ of the I-th RBF function, and one example is given In 

40 Fomiula 8. 

Formula 8 
[0131] 

45 

VP - VP_H) = II VP - VP_/||^**^>log(|| VP - VP Jj) (a) 

so ♦(! VP - VPJ|)= exp(-|| VP - VP_/f /b) (b) 

where (a) is a logistic function and k is a parameter to control the slant of the function. Also, (b) is a Gaussian function, 
and like k, b is also a parameter expressing the fomn. In addition, many others are conceivable, but since (b) is generally 
used, (b) is used here, too, with b = 1 .0. 
55 [01 32] Weight coefficient vector Vw = (w _0, w _1 ,.. ., w _N-1 (T: transposition) is decided on as follows. 

[01 33] To optimize the non-linear approximation in Formula 8, weight coeff k^ient vector Vw should be so decided on 
that with VP = VPJ, the estimated value of the frequency component is in agreement with the frequency component 
value obtained from the original image as far as possible. Here, P'J in the frequency area after enlargement corre- 
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sponds to PJ in the frequency area of the original image as shown in FIG. 15. It means that with frequency component 
position at P J as (aj, b_l), Vw that minimizes square enror function E (Vw) between frequency component F(a J, 
bj) at P J and estimated frequency component F(u J, v J) in P'J should be taken as optimum weight coefficient vector. 
[01 341 f^'i'st. estimated components F (u , v) at (u , v) are arranged from the low frequency area , and the k-th estimated 
component corresponding to frequency component position (aJ, bJ) of the original image will be given as FF(k). Then, 
matrix MP made up of frequency component vector Vy of an estimated enlarged image and RBF function is defined 
as Fonmula 9. 

Fonnula 9 

[0135] 

Vy - (FF(0),FF(1), ,FF(N 

MP^[MPJj] ' 

^dlKP.O-KP^Ojl), ^(||KP_0-KP.l||),.....^(||KP.O-KP_A^ 
^(||I^.1-KP.0|1),^(1|KP J-KP_f 0(|IW,l-KP_//-lI|) 



#(l|W>.i\^-l-KP_0|D, 0(||KP_iV-l-KP.lJ)..,., 0(||KP^Ar-l-KP_A^-l|D 

[01 36] From this, frequency component vector vy can be rewritten as Fomfiula 1 0. 

Fonnula 10 

[0137] 

Vy = MP'VW 

[0138] And the evaluation function (which represents square error function) E (Vw) Is given as In Fonnula 11. In 
Formula 11 , vector Vf Indicates that frequency components F (a, b) of an original image are arranged from the kyw 
frequency area one after another, the number of arranged components being N = n x n pieces. It is a vector that will 
be settled if an original image is given in advance. 

Formula 11 

[0139] 

E(yw) - ^^(FCa_i,b^l)-'FXuJ,vjyf 

Vf«{Fm,F(0;L),.....,FiN-l^ 
[0140] Then, if rewritten with attention paid to dE(Vw) / dVw =: 0, Vw will be given as in Fomnula 12. 
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Formula 12 
[0141] 

* T -1 

[01 42] Using this, AC deriving means 1 200 calculates approximation weight coefficient Vw. Using weight coefficient 
Vw, non-linear estimating means 1201 estimates DCT component P(u, v) at (u, v) from Fomnula 7. 

'0 [0143] In this connection, in case (b) Is used In Fonmula 8, the folbwing method of setting a value of b is conceivable. 
When an original Image is estimated - using many of image data prepared In advance - from a reduced image prepared 
in the sub-sampling, the relation is derived statistically between parameter b for the SN ratio of the estimated Image 
and the original image to be a maximum and the distribution of data within the respective original Images (standard 
deviation, average, dispersion etc.), whereby a value of b is decided on. 

IS [01 44] On the basis of the estimation results of the frequency components possessed by the enlarged image obtained 
by Rl generating means 101, Inverse orthogonal transfonn means 1213 performs inverse discrete cosine transfomi 
(IDCT), thus restoring the enlarged mage as the value In real space. Estimated Enlarged Images (EsFI) output means 
1214 - obtained by inverse orthogonal transform means 1 21 3 - to be handled by other Image processing devices, that 
is, to be displayed on CRT, to be outputted on the printer or the like. 

20 [0145] As set forth above, according to the present embodiment, the features of the frequency components of an 
original image can be approximated with precision, and it is possible to estimate the high frequency components erased 
in the sampling of the original image in a handy manner with high precision without preparing a rule etc. in advance. 

Embodiment 12 

25 

[01 46] FIG. 1 6 shows the arrangement of the image processing device of Embodiment 1 2, and there will be described 
the operation of this image processing device. As in Embodiment 1 1 , an original Image obtained by Image input means 
10 Is transfomned Into frequency component F(u, v) by Ol orthogonal transforming means 11 . At the same time, edge 
generating means 1600 extracts edges by a Laplacian filter shown in FIG. 18. 
30 [0147] Either of Laplacian filters shown in FIG. 1 8 (a) or (b) may be used for the purpose. Some other filter may also 
be used. The precision of edge extraction can be improved by selecting a filter depending on the features of the original 
image to be handled. 

[0148] For example, the Laplacian filter shown In FIG. 18 (a) multiplies the density value at object picture element 
by eight times and subtracts the density values of the eight surrounding picture elements. That is, the difference between 
35 the density value of the picture element in the center and the density value of the surrounding picture elements is 
added to the density value of the picture element in the center, whereby the picture elements that changes greatly in 
the density difference from the surrounding as in the edge are enhanced. 

[0149] Enlarged Edge (EE) approximating means 1601 in Enlarged Edge (EEd) estimating means 120B linearty 
enlarges an edge image (FIG. 1 7 (c)) -obtained by edge generating means 1 600 as shown in FIG. 17 - to a desired 

40 enlargement ratio s, and Interpolation picture elements have to be embedded between the existing picture elements 
as the Image is enlarged. Here in present embodiment, that is realized by embedding the mean value of the edge 
image data derived from the existing picture elements in the center between the existing picture elements. For example, 
in case an existing edge image is enlarged to one with picture element positions P(x _0, y _0) and Q(x _Ofdx, y _0) 
and it Is necessary to interpolate between them in the x direction according to the number of enlarged picture elements, 

45 the interpolation will be as given by Formula 13 (a). Here, it is assumed that data at picturie element position P is 
represented by D(x _0, y _0), data at picture element position Q is represented by D(x _0+dx, y .Ofdy), the position 
H to be embedded with interpolation data Is represented by H(x _h. y _h), and Interpolation data to be embedded is 
represented by DH(x _h, y _h). In case interpolation has to be done between picture element position P(x .0, y _0) 
and R(x _0, y _OHiy) in the y direction, data is as given by Fonmula 13 (b). In this connection, the interpolation data 

so position in (b) is represented by l(x J, y J) and the interpolation data Is represented by Dl(x J, y _0- 

Formula 13 
[0150] 

55 

(a) 
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DH(xJh,yJi) = (O{x^0,>La)+D(x^0 + d&r,y^O ))/2 
x_h = (x^Of x^Of dhf)/? 

5 

y_h=y_0 



(b) 

DI(xJ,yJ) = (D(x_0,y^0))+D{xJi,y_0+dy)y2 
x_/= x_0 

15 X-' = (x_0+y_(>4-d>^)/2 

[0151] Furthermore, it is considered that if interpolation is carried out in the y direction, too, using two interpolation 
picture elements neighboring each other in the y direction (obtained by interpolation in the x direction), the interpolation 
precision will further improve. 

20 [0152] Edge Frequency (EF) enerating means 1602 in EEd estimating means 120B perfomns orthogonal transform 
on an estimated image of an enlarged edge (FIG. 17 (d)) obtained by EEd approximating means 1601 to find frequency 
components (FIG. 1 7 (e)). That is done mainly t>ecause of the clearness of the image and the features of details of the 
image and also because plenty of high frequency components representing the edge are contained in such an edge 
image. In other words, it is based on the Idea that because an extracted edge image Is tacking in information on the 

25 Other portion, high frequency components appear, but lower frequency components will come out at a low level only. 
And the frequency components possessed by the enlarged image are estimated by substituting the low frequency area 
of the frequency components of the enlarged edge image (FIG. 17 (e)) obtained by EdF generating means 1602 with 
the frequency component data (FIG. 1 7 (b)) - obtained by 01 orthogonal transfomning means 1 1 - which have the whole 
features of the original image (FIG. 17 (a)). And the enlarged image (FIG. 17 (g)) brought back to the real space data 

30 through inverse orthogonal transform means 1 21 3 is outputted so as to be handled by other image processing devices, 
that Is, to be displayed on CRT, to be outputted on the printer or the like. 

[0153] That way, the enlarged image can be estimated without dlffkxilty without using an RBFN method as in Em- 
bodinfient 11. Furthemiore, the edge infonmation is handed over in the original condition, and thus the edge can be 
enhanced without losing the high frequency components of the original image and the blurring of the inr^ge can be 
35 kept down. 

Embodiment 13 

[01 54] FIG. 1 9 shows the arrangement of the vnage processing device of Emk>odiment 1 9, and there will be described 

40 the operation of the image processing device. 

[01 55] OCT transform is applied to an original image (FIG. 20 (a)) obtained by Image input means 1 0 so as to derive 
frequency component data (FIG. 20 (b)), and at the same time an edge image (FIG. 20 (c)) of the original image is 
generated by edge generating means 1600 using a Laplacian filter as shown in FIG. 18. Edge Images (El) orthogonal 
transforming means 1900 in EEd means 120B perfomns DCTon the edge image to acquire the frequency component 

45 data of the edge image (FIG. 20 (d)). 

[0156] From the frequency component data of this edge image, Edge Frequency (EdF) estimating means 1 901 in 
EEd means 1 208 estimates the frequency component data - to be obtained from the edge portion of an enlarged image 
(FIG. 20 (e)) - using radial basis function networic (RBFN) adopted in Embodiment 11 which is excellent in nonlinear 
approximation. And the tow frequency area of the frequency component data thus estimated is substituted with the 

50 frequency components obtained from the original image (FIG. 20 (f)), thereby acquiring the frequency component data 
of the enlarged Image. 

[01 57] That way, as Embodiment 1 2, the enlarged edge infomnation containing plenty of high frequency component 
data of the enlarged image is estimated from the features of the high frequency components of an image contained 
mainly in edge infonmation, and by using that, the high frequency components can be compensated well which give 
55 clearness to an enlarged Image. Furthenmore, In Embodiment 1 2, the frequency component data of an enlarged edge 
Image Is estimated from a simple llnearHnterpolated image of the edge image of an original image. Interpolation of 
present embodiment is equivalent to non-linear approximation and it is considered, therefore, that the estimation pre- 
cision in present embodiment will be higher than the method of simply perfomning linear interpolation between two 



18 



EP1 164 781 A1 



samples. 



[0158] This non-linear estimation method is different from that in Embodiment 1 1 only in that the frequency component 
data of the edge image obtained from the original Image, and not the frequency component data of the original Image, 
is inputted in RBFN of non-linear estimating means 1201 used in Embodiment 11 , and will not be explained. 
[0159] As this non-linear estimation method, it is possible to adopt a neural network technique using leaming data, 
a technique following a fuzzy mle, a chaos deterministic prediction technique etc. as described in Embodiment 11 . It 
should be noted that for those techniques, It Is necessary to prepare large-scale leaming data In advance. 
[01 60] Furthennore, EdF estimating means 1 901 finds the intemnediate value between the DCT components (points 
n - 3, n - 2, n - 1 in FIG. 21 (a)) of an image sampled from the high frequency side according to the enlargement ratio 
s as shown in FIG. 21 . And the frequency components of the enlarged edge can be estimated by allocating the DCT 
component positions starting from the head side of the high frequency components (points n-1+t-3, n-1-i-t-lln 
FIG. 21 (b)). That way, the high frequency components can be compensated that give cleamess to an enlarged image. 
[0161] It appears that technically, there is not much difference between the interpolation method shown in FIG. 21 
and that in Embodiment 12. In Embodiment 12, however, there is a possibility that the features of the edge etc. wilt 
change depending on between which picture elements of the edge image the interpolation picture element should be 
preferentially embedded when an enlarged edge image is prepared, and a proper interpolation order has to be worked 
out. But In case Interpolation is done with the frequency components as shown in FIG. 21 , the interpolation values of 
the DCT components should be embedded head side with the high frequency components since the point is to com- 
pensate the high frequency components. Then, the frequency components of the enlarged image should be estimated. 

Embodiment 14 

[0162] FIG. 23 shows the arrangement of the image processing devtee of Embodiment 14. 
[01 63] Many of original images inputted by image input means 1 0 have hundreds of picture elements x hundreds of 
picture elements. If orthogonal transform as in Embodiments 11,12 and 13 is applied to that original image at a time, 
it will take a vast amount of time. To avoid that, the data is usually divided into blocks, each a size of 4 picture elements 
X 4 pkrture elements to 1 6 picture elements x 1 6 picture elementis, and the respective blocks are enlarged to a picture 
element size according to a desired image enlargement ratio s, and they are put together again. In the case of this 
method, however, the following problem are pointed out. 

[01 64] FIG. 22 (a) schematbally shows that problem. Forthe purpose of stmptrcity, there will be explained an example 
where one-dimensional DCT is applied to the density value D(x, y.O) of the x-direction pteture elements with y = y .0 
in block Ai (shown In solki line). 

[0165] There are 0 to n-1 pieces of digital image data (density values) in the x direction. To perform one-dimensional 
DCT on them is synonymous with approximating with a linear combination of cosine function cos (i x n/n) (1 = 0,...., 
n-1 ) having n cycles, assuming that a function expressing n pieces of image data Is a cyclic f uneven having n cycles 
in the x direction. Accordingly, D(x, y_0) can be expressed as in Formula 14. 



[0167] aj represents the i-th DCT component. In such a case, as shown in solid line in FIG. 22 (a), it is presupposed 
that in section [n, 2n - 1], the same data as in section [0, n - 1] is repeated, and therefore, image data D(n, y_0) with 
x = n will try to detemnine aJ so as to be identical with data D(0, y_0) with x = 0. Therefore, the larger the difference 
between data D(n - 1 , y_0) with x = n - 1 and data D(n, y_0) = D(0, y_0) with x = n, that is, x = 0, the wider the gap 
between the enlarged data D embedded in the x direction and data D (n, y_0) with x = n of the enlarged data obtained 
from the from the next block at x = n to x =2n - 1 . As a result, D (x, y_0) will be discontinuous In the x direction near 
this border. 

[0168] Since two-dimensional DCT is one obtained by expanding one-dimensional DCT in the y direction, that dis- 
continuity will also occur when the one-dimensional way is expanded to two-dimensional DCT. The present embodiment 
is a process addressing this problem. 

[0169] An example will be considered where an original image will be divided into (n x n)-sized blocks Ai (i = 0,..., L 



Formula 14 



[0166] 
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- 1) and each block is enlarged to (m x m)-sized blocks Bi (i = 0...., L - 1) thereby enlarging the original image at an 
enlargenfient ratio s = m/n. L corresponds to the number of blocks. Block dividing means 2300 in FIG. 23 divides the 
original Image into blocks of not a size of n x n but blocks Al' (i = 0,...,L-1) of a size of (n + u) x (n + u) so that the 
neighboring blocks partly overiap each other as shown on the left in FIG. 24. 
5 [0170] That is, as shown in FIG. 22 (b), OCT on block AO is perfomned on section [0, n + u] (in this case, it is pre- 
supposed that in section [n + u 1 , 2n + 2u], the same data as in section [0, n + u] will be repeated (shown in dotted 
line in FIG. 22 (b)). Similariy, DCT on block Al is perfomned on sectk>n [n, 2n + u - 1]. That way, it is possible to keep 
down the gap in density value occurring in the block border when the conventional method is used (see frame R in 
FIG. 22). 

10 [0171] In other words, noise N is caused in the end portbn of block AO, and the data in this portion of block AO is 
not adopted but the data in A1 is used. 

[01 72] Then, EBl frequency means 2302 enlarges the frequency component data of block Ai to frequency component 
data of ((n + u) x s) x ((n + u) x s). This enlargement is the same as that In Embodiments 1 , 2 and 3 except that the 
enlargement is carried out block by block in the present embodiment, and will not be explained. The block size n x n 
IS that is used is generally 4 pk^ture elennents x 4 picture elements to 1 6 picture elements x 16 pbture elements. If n = 
8, u that determines the overlapping portion is set at 3, but this is not restrictive. 

[01 73] Block frequency extracting means 2303 does not use all the frequency component data of block Bi' obtained 
but takes out data of the required size m x m from the low frequency side, and again processes it to make an enlarged 
block Ci. And on Ci, block inverse orthogonal transfonn means 2304 effects orthogonal transfomi, and Enlarged Images 
^ (El) recomposing means 2305 places the image data generated from block Ci at the corresponding position, thus finally 
obtaining an enlarged image. 

[01 74] That way, it is possible to avoid discontinuity between the interpolation data of block Ci near the border of the 
neighboring blocks and the image data at the head of the next block Ci + 1 . 

25 Embodiment 15 

[01 75] FIG. 25 shows the arrangement of the image processing devbe of Emk>odiment 1 5. In FIG. 25, Block (B) data 

transforming means 2500 perfonris a specific function transfomi - which will be described later - on the image data 
within blocks Ai (i = 0,..., L - 1 ) of an image size of n x n into which the original image is divided by block dividing means 
30 2300 so as to keep down distortion occurring in the block image after enlargement. Other than that, the present em- 
bodiment is identical with Emk)odiment 14 in arrangement except that the present embodiment drops B frequency 
extracting means 2303 adopted in the image processing devk:e in Embodiment 14. 

[0176] In the Image processing device according to Embodiment 14, the original innage is divided by block dividing 
means 2300 into blocks of a little larger size than the size of n x n so that the respective blocks overiap, and frequency 

35 component data of a desired size is taken out of an enlarged block obtained and substituted again with the frequency 
data of the enlarged block whose original size is n x n. In the present embodiment, on the other hand, bkx:k dividing 
means 2300 divides the original image into bk)cks Ai (i = 0, .... L - 1) of a size of n x n without having them overiap as 
shown in FIG. 26. And data within bkxk is transfomied as will be described later, and enlarged blocks Bi (i = 0,..., L- 
1 ) are estimated from the transformed image data. That is the point where the present embodiment is different. 

40 [0177] There will be described the operation of the image processing device thus constructed of Embodiment 15. 
An original image is divided into blocks, and image data in the respective blocks Ai (i = 0, .... L - 1) are transformed by 
B data transforming means 2500 as follows. 

[0178] FIG. 27 (a) shows the transition of density value D(x, y) in blocks Al. And the diagram of Fig^7(b),(c)and(d) 
show the transition D(x, y_0) in the x direction at y = y_0. 
^ [0179] First, function p(x, y_0) that passes through image data D(0, y_0) at x = 0 and image data D(n - 1 , y_0) at x 
= n - 1 will be defined. There are many function examples, but here a one-dimensional formula as in Formula 15 will 
be defined (FIG. 27(b)). 

Fomnula 15 

50 

[0180] 

P (X, y^O) = D(0, yj)) + (D(n - 1, y_0) - D(0, yj>)yn - x 

55 

[0181] This p(x, y_0) will be subtracted from D{x, y_0). That is, <l>(x, y_0) = D(x, y_0) • p(x, y_0) will be calculated 
(FIG. 27(c)). This calculation is made for each y_0 within the block. In the y direction with x = x _0 fixed, furthemiore, 
a simitar subtraction function <I>(x_0, y) = D(x_0, y) • p(x_0, y) is worked out. And image data is normalized with the 
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maximum value Max O at (x = 0,..., n - 1 , y 0,..., n - 1 ) of absolute values of those subtraction function (FIG. 27 (d)). 
[01 82] Because data at this moment, seen In the x direction, is made such that data at the block border x = n - 1 and 
X = 0, that Is, x = n are all 0 and all of density values between n - 1 and 0, the difference between interpolation data to 
be embedded after x = n - 1 and data at x = n is very small. Thus, the distortion by block connection can be kept down. 
5 [0183] Here in this example, a one-dimensional expression as Fonmula 15 Is used. But generally any functions that 
satisfy the border conditions at points G and H in FIG. 27 are no problems, and therefore if a function that can keep 
down fluctuation after transform Is used for p(x, y_0), distortion can be furthennore suppressed and the enlarged 
frequency components can be estimated with precision. 

[0184] As set forth above, by bringing close to the density value at the block border for every block of the original 
10 Image data, it is possible to reduce the discontinuity of the block border that causes fluctuation in estimating the enlarged 
image perfonned by Interpolation on orthogonal transform. 



Embodiment 16 

IS [0185] FIG. 28 shows the arrangement of the image processing devk^ according to Embodiment 16. 

[0186] In estimating an enlarged image of a color original image, the problem is that enlargement will be a vast 
amount of processing as compared with the enlargement of one^lor multiple gradation data. The present embodiment 

is an invention to make the processing efficient. 

[0187] In a color original image Inputted by image input means 10, SC selecting means 2800 selects a color com- 
20 ponent to be made a standard. Generally, the color original image is made up of three colors - red, green and blue. 
Considering that green data is much reflected on luminance infomnation, it is desirable to select a green component 
as standard component. TR depriving means 2801 finds simple ratk> ratio.r of red component to green component 
and simple ratio ratio _b of blue component to the green component. There are a variety of methods of finding the 
simple ratio. To be used here in this example are the mean value of density ratios to green of red within the object area 
25 and the mean value of density ratios to green of blue within the object area as in Fomiula 1 6. 



Formula 16 
[0188] 

30 



ratio 



35 n-j 

ratio 



[0189] In Formula 16, rjj, gL.ij, bjj represent the densities of the red, green and blue components respectively at 
40 picture element position (i, j) of an original image. Instead of estimating all the remaining components of the object 
area using one ratio coefficient like that, it is also possible to adopt matrix R_r made up of the ratio of the red component 
to the green component in each picture element and matrix R_b made up of the ratio of the blue component to the 
green component In each picture element. This way, it is possible to reproduce the features of the color original Image 
better and enlarge the color image with higher precision than when using one ratio coeffteient. 
45 [0190] On this standard component or the green component, as in Embodiments 11 to 15, enlargement is performed 
by Standard Images (SI) orthogonal transfonning means 2802 transfomi, Standard Enlarged Image Frequency (SEIF) 
estimating means 2803 and Standard Inverse (SI) orttiogonal transfonning means 2804. And ShC enlarging means 
2805 multiplies the simple ratio ratio_r, ratio.b by the enlarged green data from SI orthogonal transforming nneans 
2804, thus producing enlarged data of the red and blue components. These three enlarged components are combined 
so into one to produce an enlarged Image of the color original image, and EsEI output means 1214 outputs the data to 
be handled by other image processing devices, that is, to be displayed on CRT, to be outputted on the printer or the like. 
[0191] By perform orthogonal transform of one component alone, it is possible to save the trouble of enlarging each 
of a plurality of components making up a color original Image and to simplify the processing. 



S5 Embodiment 1 7 



[01 92] In Embodiments 1 7 to 21 , there will be explained about Image processing devtees to enlarge an Image using 
Wavelet transform. 
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[0193] FIG. 29 shows the arrangement of the innage processing device of Embodiment 17. image input means 10 
is the same as those in Embodiment 1, Embodiment 11 and others. The II regulating means 12 Interpolates or thins 
out (hereinafter both expressed as "regulate") the horizontal, vertical picture elements of an original image having n 
picture elements x n picture elements obtained by image input means 10 to 1/2 of a desired enlarged image size of 

s Ln picture elements x Ln picture elements. Image enlarging means 290A enlarges the Image using multiple resolution 
analysis in Wavelet transfonn which wilt be described later. Enlarged Images (El) regulating nrteans 2913 regulates to 
a desired image size of Ln picture elements x U\ picture elements an enlarged image which was regulated by II reg- 
ulating means 12 and having four times as many picture elements as the original image. EsEI output means 1214 
outputs an Image data after enlargement - estimated by El regulating means 2913 - to other devices as for display. 

10 [0194] Image enlarging means 290A Is provided with Vertical Edge (VdE) generating means 2900 which takes out 
an vertical-direction edge component image from an original Image regulated to Ln/2 picture elements x Ln/2 picture 
elements by II regulating means 12, Horizontal Edge (HEd) generating means 2901 which takes out an horizontal- 
direction edge component image and Oblique Edge (OE) generating means 2902 that takes out an oblique-direction 
edge component image. Furthermore, image enlarging means 290A has leveling up means 2903 which generates an 

IS enlarged image of Ln picture elements x Ln picture elements by Inverse Wavelet transfonn by regarding the above- 
mentioned three edge component Images and an original image regulated to Ln/2 picture elements x Ln/2 picture 
elements regulated by II regulating means 12 as four sub-band components making up an transfomned Image at the 
time when the enlarged image of Ln picture elements x Ln picture elements was subjected to Wavelet transfonn. 
[01 95] In the thus f onned image processing devtee of Embodiment 1 7, multiple resolution analysis by Wavelet trans- 

20 form is utilized for image enlargement. Wavelet transform, whbh is described in a number of publications including 
"Wavelet Beginners Guide," Susumu Sakakibara, Tokyo Electric Engineering College Publk^tion Bureau, is devetoped 
and applied in many fields such as signal processing and compression of image data. 

[0196] An transfomned Image of an original image, whbh is obtained by subjecting the original image to Wavelet 
transform, is formed of a number of sub-band (partial frequency band) components. FIG. 31 shows a layout example 
25 of sub-bands of a Wavelet transfomned image In which an original Image Is divided into 1 0 sub-bands, LL3, HL3, LH3, 
HH3, HL2, LH2, HH2, HL1, LH1, HL1. 

[0197] FIG. 30 Is a diagram in whteh Wavelet transfonn as shown In FIG. 31 is illustrated in the form of filter series. 
That is, the Wavelet transfonn is perfomned in three stages - stages 1, 11, III. In each stage, "Low" processing and "High" 
processing are performed in the vertical direction (y direction) and horizontal direction (x direction) separately. In the 
30 Low processing, low pass filtering and down-sampling (thinning out) to 1/2 are canied oiit. In the High processing, high 
pass filtering and down-sampling to 1/2 are conducted. 

[0198] First, High processing and Low processing are perfonned on the original image in the horizontal direction. 
And on the output of the horizontal direction High processing, High processing and Low processing are perfonned In 
the vertical direction. The result of the High processing the vertical direction is the HH1 component, and the result of 
35 the Low processing In the vertical direction is HL1 . Then, on the output of the horizontal-direction Low processing, Low 
processing and High processing are perfonned in the vertbal direction. The result of the vertk;al-direction Low process- 
ing is LL1 and the result of the vertical-direction High processing is LH1 . Those are the results obtained In the first 
Wavelet transforming. 

[0199] Further, Low and High processing in the horizontal direction is applied to the LL1 component. And to the 

40 output of the horizontal direction High processing is applied in the vertical direction, and the result is HH2. To the output. 
Low processing is also applied in the vertbal direction, and the result is Ht-2. Further, to the output obtained by the 
Low processing in the horizontal direction, Low processing is applied in the vertical direction, and the result Is LL2. To 
the output, High processing is also applied in the vertk^ direction, and the result Is LH2. Those are the results obtained 
in the second Wavelet transfonn Ing. 

45 [0200] Likewise, the LL2 is subjected to horizontal-direction Low processing and High processing separately. In the 
vertk:al direction, too, Low processing and High processing are performed separately. Thus obtained are the sub-band 
components HH3, HL3, LH3, LL3. Those are the results obtained in the third Wavelet transfomning. 
[0201] In the first stage of the Wavelet transfonn as shown, the original image is broken down into four frequency 
components - LL1 , HL1 , LH1 and HH1 and down-sampled to 1/2 both in the horizontal and vertk:al directions. Therefore, 

so the size of the image representing the respective components will be 1/4 of that of the original image. LL1 Is a low 
frequency component extracted from the original image and is a blunred image of the original image. Most of Infonnation 
of the original image is contained in that component. Therefore, LL1 is the object for the second Wavelet transform. 
[0202] Meanwhile, the HL1 component obtained by the processing in FIG. 30, represents an image with the high 
frequency component extracted intensively In the horizontal direction of the original image. The LH1 component rep- 

55 resents an Image with the high frequency component extracted Intensively In the vertical direction of the original Image. 
And HH1 represents an Image with the high frequency component extracted both in the horizontal and vertical direc- 
tions. In other words, it can be conskiered to be an image with the high frequency component extracted In the oblique 
direction. 
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[0203] Viewed from the density vaiue, it Is considered that the HL1 component strongly reflects the area where the 
density value fluctuates violently in the horizontal direction of the original image (edge infomiation in the vertical direc- 
tion). On the other hand, LM1 component strongly reflect the area where the density value fluctuates violently in the 
vertical direction of the original image (edge information in the horizontal direction). The HH1 component strongly 
5 reflects the area where the density value fluctuates violently In the horizontal and vertical directions of the original 
image (edge Information In the oblique direction). 

[0204] Such characteristics produced fcty Wavelet transf onu can also be said of the components LL2, Hl-2, LH2, IHH2 
obtained in the second stage of Wavelet transfomi of LL1. The same is applicable to the components of Wavelet 
transfomi with LL2 as object image. Like this, it can be taken that the Wavelet transfomn breaks the Ll image with the 
10 low frequency component extracted In the sub-band component image of one stage before down into four 1 /4 resolution 
Images corresponding to the low frequency component and the frequency components in vertical, horizontal and ob- 
lique directions. 

[0205] And these sub-band component images can be synthesized by filtering to restore an image of one stage 
before. This will be explained in FIG. 31 . Synthesizing four sub-band component images LL3, HL3, LH3, HH3 can 
15 restore LL2, and synthesizing Ll^, HL2, LIH2 and I-IH2 restores U.1 . And the original image can be restored by using 
LL1,HL1,LH1.HH1. 

[0206] Because such a Wavelet transfomn can express a plurality of sub-band component Images with different 
resolutions simultaneously, It is also called the multiple resolution analysis. And the Wavelet transfomi attracts attention 
as technique that can compress data efficiently by compressing the respective sub-band components. 

20 [0207] In the present embodiment, first, the original image is regarded as the low frequency sub-band component 
LL1 at one stage before. The next step is to estimate the remaining images - image HL1 with the high frequency 
component strongly extracted In the horizontal direction, image U-il with the high frequency component strongly ex- 
tracted in the vertical direction, and Image HH1 with the high frequency component strongly extracted in the horizontal 
and vertical directions, and to obtain an enlarged image four as large. And this processing is applied to enlarge an 

25 original image to a desired size. 

[0208] On the basis of the above description , the operation of the image processing devbe of the present embodiment 
will be explained. 

[0209] First, image input means 1 0 reads out an original image of a size n pk:ture elements x n picture elements to 
be enlarged. 

30 [0210] The original image read by image input means 10 is regulated in image size by II regulating means 12. As 
mentioned in the description of the multiple resolution analysis in Wavelet transfonm, if an image, an object for transform, 
is subjected to a Wavelet transform, the sub-band components after transfomn will always become 1/2 of the original 
size both In the number of picture elenrents in the horizontal direction and the number of picture elements in the vertteal 
direction. On the other hand, the picture element size obtained by inverse Wavelet transfomn will be twee as large as 

35 that of the original sub-band component both in the number of pbture elements in the horizontal direction and the 
number of picture elements in the vertical direction. That is, the total number of picture elements will be four times as 
many. 

[0211] In view of such a nature, it is desirable that the number of picture elennents both in the horizontal and vertk^al 
directions of an enlarged image obtained in an inverse transfomn is a multiple of 2. For this reason, II regulating means 

40 12 first regulates a desired enlarged image size Ln picture elements x Ln pk:ture elements - the enlargement ratio 
being L-to a multiple of 2, that is, dLn picture elements x din picture elements and regulates the original Image so that 
its size is dLn/2 picture elements x dLn/2 picture elements. A number of regulating techniques are available, but here 
in this embodiment It is to be realized by Interpolating between the picture elements using Formula 1 or by thinning 
out prcture elements in areas where the gradation less change. 

45 [0212] Here, the following methods are also possible to apply. That is, the original image is transfomied into frequency 
space by orthogonal transform such as OCT transfomi, and the frequency components corresponding to dLn/2 picture 
elements x dLn/2 picture elements are taken out. Or the shortage high frequency components are emt>edded with "0" 
and regulated by inverse orthogonal transform corresponding to dLn/2 picture elements x dLn/2 picture elements. But 
considering the processing effk^iency and enlargement by Wavelet transform tectinlque after that, It is not thought that 

50 a complicated processing Is so e;fflc:ient. For this reason, simple-interpolating between pteture elements or thinning out 
is adopted. 

[021 3] Image enlarging means 290A enlarges an original image-regulated to dLn/2 picture elements x dLn/2 picture 
elements by II regulating means 12 - twice both in the horizontal and vertk:al directions to an image size close to a 
desired Ln pkture elements x Ln picture elements. For this, the multiple resolution analysis in the Wavelet transfomn 
55 is utilized. The problems with the prior art method in which orthogonal transfomn is perfomned and shortage component 
is compensated in the frequency area are processing time and occurrence of jaggy noise in block joint because an 
image is divided into a plurality of blocks. But the Wavelet transfomn offers an acivantage that because a large Image 
c:an be handled at a time, no such noise will be caused. 
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[0214] Furthermore, in case sub-band component LL1 in FIG. 31 is taken as image regulated to dLn/2 picture ele- 
ments X dLn/2 picture elements, image enlarging means 290A will have to estimate the images of dLn/2 picture ele- 
ments X dLn/2 picture elements corresponding to the remaining three sub-bands HL1 , LH1 , HH1 . 
[0215] FIG. 32 schematically shows that procedure. Here, a method is adopted in which the three images of sub- 

5 band components I-IL1, LH1, HH1 are talcen as edge images of LL1 in three directions. /Vs mentioned above, the 
component HL1 Is to represent an image with the high frequency component strongly extracted in the horizontal di- 
rection of an enlarged image (will be named LLO) having four times as many picture elements as the original image 
regulated to dLn/2 picture elements x dLn/2 picture elements (FIG. 31 (a)), and the sub-band component Ll-i1 is to 
represent an image with the high frequency component strongly extracted in the vertical direction of sub-band com- 

10 ponent LLO. And the sub-band component HH1 will be an image with the high.f requency extracted both in the horizontal 
and vertical directions. That is, it is considered, the sub-band component HL1 reflects the area representing a high 
frequency component in the horizontal direction, that is, edge information in the vertical direction of the image of sub- 
band component LLO (FIG. 32 (b)). On the other hand, It is considered, the sub-band component LH1 reflects the area 
representing a high frequency component in the vertical direction, that is, edge infomiation in the horizontal direction 

IS of the image of sub-band component LLO (FIG. 32 (c)). And it is considered, the sub-band component HH1 reflects 
-the area representing a high frequency component both in the vertical and horizontal directions, that is, edge infomiation 
in the oblique direction of the image of sub-band component LLO (FIG. 32 (d)). 

[0216] In the present embodiment, therefore, there is provided edge generating means 290B of an arrangement as 
shown in FIG. 29 where VdE generating means 2900 extracts the edge component in the vertical direction of an original 

20 image regulated to dLn/2 picture elements x dLn/2 picture elements by II regulating means 1 2 and taltes it as shortage 
HL1 component. Aj\6 HEd generating means 2901 extracts the edge component in the horizontal direction of the 
original image regulated to dLn/2 picture elements x dLn/2 picture elements by II regulating means 12 and takes it as 
shortage LH1 component. Similarly, OEd generating means 2902 extracts the edge component in the oblique dire^ion 
of the original image regulated to dLn/2 picture elements x dLn/2 picture elements by II regulating means 12 and takes 

25 it as of shortage HH1 component. 

[0217] In this process, it is to be assumed that the edge generating means 2900, 2901 and 1902 use edge detection 
filters to perform detection in three directions as shown in FIG. 33. An example shown in FIG. 33 (a) is to detect the 
edge in the horizontal direction using a filter in whk:h weighting increases in the horizontal direction. An example shown 
in FIG. 33 (b) is to detect the edge in the vertical directton using a filter in whk:h weighting increases in the vertical 

30 direction. An example shown in FIG. 33 (c) is to detect the edge in the oblique direction using a filter in which weighting 
increases in the oblique direction. It is not that these filters only are applbable, but other filters may be used. 
[021 8] With the estimated four sub-band components subjected to inverse Wavelet transf onn as described, leveling 
up means 2903 acquires a dear enlarged Image of a size dLn picture elements x dLn pk^ture elements. In this case, 
since the Wavelet transform can be illustrated in filter series processing as shown in FIG. 30, the processing here may 

35 be filter series processing that does the reverse of that. 

[0219] If the size of an enlarged image obtained by image enlarging means 290A, that is, the desired image size Ln 
picture elements x Ln pbture elements Is not a multiple of 2 (In case Ln/2 is not an integer), a delicate difference occurs. 
In such a case, El regulating means 2913 does interpolatk>n between picture elements or thinning out to make up for 
the delicate difference. To be processed here is at most one picture element, and therefore the processing is done in 

40 the area where the image changes are small (areas - other than the edge - where there Is small gradation change) 
and its effect is small. 

[0220] An enlarged image obtained by El regulating.means 291 3 is handed over to other devnes by Enlarged Images 
(El) output means 2914 and displayed on CRT or used in some other way. 

[0221] According to the present embodiment, as set forth above, the blurring of an image can be kept down unlike 
45 the prior art method of merely interpolating between the picture elements in the original image and the prior art devk:e 
in which the shortage of frequency components in the frequency area are embedded with "0." The present embodiment 
also produces a clear enlarged image without causing noise like Jaggy etc. which Is considered a problem encountered 
with the orthogonal transform method. According to the present embodiment, furthenmore, it is possible to estimate an 
enlarged image in a handy manner without preparing a rule in advance. 

50 

Embodiment 1 8 

[0222] FIG. 34 is a block diagram showing the arrangement of image enlarging means 290A making up the image 
processing device of an eighteenth embodiment 18 of the present invention. The operation of this devtee will be ex- 
55 plained. 

[0223] /Ks in Embodiment 1 7, an original image obtained by image input means 1 0 is regulated by II regulating means 
12 from a desired enlarged image size Ln picture elements x Ln picture elements to dLn/2 pk:ture elements x dLn/2 
picture elements, both a multiple of 2, in the horizontal and vertk:al directions. And the original image is regulated to 
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a 1/4 size, that is, dLn/2 picture elements x dLn/2 pbture elements both in the horizontal and vertical directions. 
[0224] Receiving the results of the processing by II regulating means 12, input fine-adjustment means 700 fine- 
adjusts dLn/2 picture elements x dLn/2 picture elements by one picture element to a multiple of 2, that is, ddLn picture 
elements x ddLn picture elements so that the sub-band component one level below is acquired at leveling down means 
5 701 from the original image regulated to dLn/2 picture elements x dLn/2 picture elements. /Vnd leveling down means 
701 performs Wavelet transform on the original image of ddLn picture elements x ddLn picture elements acquired at 
input fine-adjustment means 700 and generates four sub-band components LL2, HL2, LH2, HH2 with the Image size 
being 1/4 of that of the original image. 

[0225] FIG. 35 schematically shows the outline of the processing at image enlarging means 290A. 

10 [0226] In Embodiment 17, the edge image in the vertical direction, the edge image in the horizontal direction and 
the edge image in the oblique direction, which were obtained from a current object image LL1 , are talcen as the sub- 
band components HL1, LH1, HH1 that are short in the Wavelet transfonmed image of the image LLO obtained by 
enlarging the current object image by four times. But strictly spealdng, this is not applicable to filtering in FIG. 30. 
[0227] For example, in case sub-band component IHLI Is acquired from sub-band component LLO by filtering, high 

15 frequency component data in the horizontal direction and low frequency component data in the vertical direction are 
extracted as HL1 component by filtering in FIG. 30. Because of that, in the HL1 component, there are extracted a 
pictu re element portion where the value fluctuates violently In the horizontal direction (edge etc. extending in the vertical 
direction) and a picture element portion where the value fluctuation is small in the vertical direction. In Embodiment 
17, it is thought that of those portions, the picture element portion where the value change is great in the horizontal 

20 direction, that is, the edge portion extends in the vertical direction has a great effect, and edge information in the vertical 
direction only is taken as HL1 component. In some images handled, there are cases where the effect of the image 
portion In which the value change is srnaW in the vertical direction can not be ignored. Furthemnore, edge information 
in the vertical direction contains plenty of the picture element portion where the value change is great in the horizontal 
direction but, strictly speaking, that is not always true. This is also applicable to other sub-band components LH1 , HH1 . 

25 [0228] In consideration of that, in the present embodiment, it is decided to prepare sub-band components LL2, HL2, 
LH2, HH2 by Wavelet-transfomiing the current object image and thus lowering the components by one level. And the 
correction amounts dHL, dLH, dHH for estimating the sub-band conriponents corresponding to the three edge images 
of the original object image are found from the correlation among the edge images HLe, LHe, HHe in three direction 
of the low frequency component LL2 in three directions in those sub-band components and the actual three sub-band 

30 components HL2, LH2, HH2. 

[0229] First, there lis provided an arrangement formed of Reference Components (RC) generating means 70A, cor- 
rection estimating means 70B, and component estimating means 70C as shown in FIG. 34. And means 702 for gen- 
erating reference HL component detects edge infonnation in the vertical direction using a titter as shown in FIG. 33 
(b) with attention paid to the LL2 component that is present in the low frequency area and expresses the features of 

35 the original image more suitable than the sub-band components LL2, HL2, HH2. That edge information will be named 
reference HL component HLe. HL correction estimating means 705 checks the comelation between the reference HL 
component HLe and HL2 obtained by leveling down means 701 . 

[0230] There are a number of methods for finding that correlation. Here, difference image dHL between reference 
HL component HLe and actual HL component HL2, that is, dHL = HLe - HL2, will be found as shown in FIG. 35 (c). 

40 Means 703 for generating reference LH component and means 704 for generating reference HH component, too, select 
edge infomnation in the horizontal direction of the LI-2 component as reference LH component LHe and edge information 
in the oblique direction of the LL2 component as reference HH component. LH correction estimating means 706 finds 
the difference image dLH = LHe - LH2 between reference LH component LHe and actual LH component LH2. HH 
correction estimating means 707 finds the difference dHH = HHe - HH2 between the reference HH component HHe 

45 and actual HH component HH2. 

[0231] And as in Embodiment 1 7, HL component estimating means 708, LH component estimating means 709 and 
HH component estimating means 710 firstly enlarge the image of correction components dHL, dLH, and dHH to the 
components with ddLn picture elements x ddLn picture elements. Next, means 708, 709 and 71 0 subtract the conrection 
components dHL, dLH, and dHH from above-mentioned edge components of HL1 , LH1, HH1 obtained by VEd gen- 

50 erating means 2900, HEd generating means 2901 and OEd generating means 2902 respectively, and estimates HL1 , 
LH1 , HH1 components at the time when the original image regulated to ddLn picture elements x ddLn picture elements 
by input fine-adjustment means 700 is taken as sub-band component LL1 . HL component estimating means 708, LH 
component estimating means 709 and HH component estimating means 710 do fine-adjustment by interpolating be- 
tween the respective picture elements in accordance with Fomiula 1 so that the picture element size of each corrected 

55 Image will be ddLn picture elements x ddLn pkrture elements when the above-mentioned con-ection components dHL, 
dLH, dHH are used. But this is not the only way, but other methods are possible to apply including a conventional 
metiiod of enlarging the image size twk:e both in the horizontal direction and the vertical direction by embedding the 
shortage component with 0 in the area where the frequency is transformed. 
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[0232] But it takes long to embed 0 in shortage areas and to perform enlargement by inverse Wavelet transfomi 
using the sub band component. In consideration of processing efficiency, simple interpolation is better and could have 
little effect on the picture quality of the finished enlarged image. For this reason, IHL component estimating means 708, 
LIH component estimating means 709 and l-il-l component estimating means 710 are to adopt a linear interpolation 
5 method as in Formula 1 . 

[0233] Processing step by leveling up means 2903 and after that is the same as in Embodiment 1 7. 
[0234] In the above description, estimation by HL component estimating means 708, LH component estimating 
means 709 and HH component estimating means 71 0 is made by adding the difference components between reference 
components and actual components. In addition to that, the following methods are well suitable. 

10 

(1 ) The correction amounts obtained at HL correction estimating means 705, LIH correction estimating means 706 
and HH correction estbnating means 707 are multiplied by a transfomri coefficient matrix. The products are added 
to the component values of VdE generating means 2900, HEd generating means 2901 , and OEd generating means 

2902 respectively. 

15 (2) The correction amounts obtained by IHL correction estimating means 705, LIH correction estimating means 706 

and HIH correction estimating means 707 are transfomied by the transform function. The results are added to the 
results of VdE generating means 2900, HEd generating means 2901 and OEd generating means 2902 respectively. 

(3) A neural network model is used which has so learned as to input in the neutral networi< the results of HL 
connection estimating means 705, LH correction estimating means 706, HH correction estimating means 707, and 

20 VdE generating means 2900, HEd generating means 2901 , and OEd generating means 2902 and to output the 

estimated values of HL, LH, HH. 

(4) The HL, LH, HH components are estimated from a large amount of data base or rule base prepared in advance 
by inputting the results of HL correction estimating means 705, LH correction estimating means 706, HH correction 
estimating means 707, and VdE generating means 2900, HEd generating means 2901 , and OEd generating means 

25 2902. 

[0235] According to the present embodiment, shortage sub-band connponents, especially high frequency compo- 
nents in Wavelet transfonm images, - which can not be taken out merely by edge detection in three directions of a 
regulated original image In Embodiment 17 - can be estimated with high predsion and thus the blunlng of an image 
30 can be kept down. Furthemnore, Wavelet ti-ansfomi does not require block division as In orthogonal transform and there 
arises no block distortion, whk;h is the problem encountered with the prior art method using orthogonal transfomi. 

Embodiment 1 9 

35 [0236] FIG. 36 shows the arrangement of image enlarging means 290A of the image processing device of Embod- 
iment 19, and there will be described the operation of this image processing device. 

[0237] The processing steps by image input means 1 0 and II regulating means 1 2 Is the same as that in Embodiment 

17, and the processing by input fine-adjustment means 700 is also identical with that in Embodiment 18. 

[0238] Reference Components (RC) generating means 3601 finds reference components to determine the respective 

40 correction amounts whbh are used in estimation - of HL, LH, HH components - from LL2 in the low frequency area in 
a sub-band component image obtained by leveling down means 701 -the estimation made by HL component estimating 
means 708, LH component estimating means 709 and HH estimating means 71 0. Here, a Lapladan fitter as shown in 
FIG. 1 8 (a)(b) is used, and the typical edge image of LL2 will be taken as the reference component image. The Lapladan 
filter is often used for detection of edges where not so much restriction is imposed on the direction, and not edges in 

^ a specific direction, as explained in Fig 33. 

[0239] With this edge as reference component image, the present embodiment finds the correction amount as in 
Embodiment 18. This way, the repeating of the edge detection procedure as pointed out in Embodiment 18 can be 
reduced, and thus the processing is made effident. 

[0240] As in Embodiment 18, HL correction estimating means 705, LH correction estimating means 706 and HH 
so correction estimating means 707 find the respective difference images dHU2, dLH2, dHH2 between the edge image 
obtained by RC generating means 3601 and HL2, LH2, HH2 obtained by leveling down means 701 (see FIG. 35 (c)), 
and each difference image is regulated to an image having ddLri picture elements x ddLn pk:ture elements by linear 
approximation as in Fonnuta 1 . 

[0241] Meanwhile, edge generating means 3600 detects an edge image using a Lapladan fitter from an original 
55 image of ddLn picture elements x ddLn picture elements regulated by 11 regulating means 12. And HL component 
estimating means 708, LH component estimating means 709 and HH component estimating means 71 0 add con-ection 
images obtained by HL correction estimating means 705, LH con-ection estimating means 706 and HH con-ection 
estimating means 707 to the edge image whereby the respective sub-band components HL1 , LH1 , HH1 can be esti- 
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mated with high precision. 

[0242] That way, it is possible to clearly estimate an enlarged Image of (2 x ddLn) picture elements x (2 x ddLn) 
picture elements. If interpolation and thinning out of a number of picture elements are added to that, an enlarged Image 
having a desired size Ln picture elements x Ln picture elements can be obtained with high precision. 
5 [0243] In addition to estimation by HL component estimating means 708, LH component estimating means 709 and 
HH component estimating means 71 0 as explained in Embodiment 1 8, it is possible to use the product of the connection 
amounts - by HL correction estimating means 705, LH conrection estimating means LH corec^on estimating means 
706, HH correction estimating means 707 - multiplied by a certain transfonn coefficient matrix or the results of transfonm 
by the transform function. 

10 

Embodiment 20 

[0244] FIG. 37 shows the an'angement of the image processing device of Embodiment 20. 
[0245] The outline of the present embodiment is this. The number of picture elements to be enlarged is not known 
15 in advance, and an original Image Is enlarged twice both in the horizontal and vertical directions in accordance with 
multiple resolution analysis by Wavelet transfonm, and the enlarged Image is shown to the user. And this process will 
be repeated until the user finds a desired one. 

[0246] , An original Image of a size n picture elements x n picture elements inputted by image input means 1 0 is set 
as an enlargement object image by EP initializing means 3700, Then, Obi enlarging means 3701 enlarges the original 
20 image of a size n picture elements x n picture elements twice both in the horizontal direction and the vertical direction, 
that is, by four times. In this enlargement step, the enlargement object image can always be enlarged to a size of four 
times as many picture elements by using image enlargement means in the image processing devices described in 
Embodiments 17, 18 and 19. 

[0247] Enlarged image presenting means 1302 shows to the user the current enlarged Image obtained by Obi en* 
25 larging means 3701 on CRT etc. Providing a function of moving the visual point with a cursor etc. if the resolution of 
the image is exceeded that of CRT etc., a function of cutting out a specific part from the image would help the user to 
Judge If the displayed enlarged Image is just a needed one. 

[0248] Receiving the Instruction from the user, MP ending judge means 3703 refers the process to Image fine-ad- 
justment means 3704 If the Image Is of a desired size, and, if an indication is received that the size of the enlarged 
30 image is not a desired one, sets this enlarged Image for next enlargement object image and retums the process to Obi 
enlarging means 3701 . 

p)249] Image fine-adjustment means 3704 asks the user If fine-adjustment Is needed. Since multiple resolution anal- 
ysis by Wavelet transfomn is used in image enlargement, the enlarged Image Is always four times as large as the Image 
before the enlargement The user may think that while the previous image is too small, the enlarged image is too large 
35 to display on CRT at a time. Image fine-adjustment means 3704 asks the user if the image size should be adjusted to 
some extent. If the user wants the image size to be enlarged a little, pbture element interpolation is perfonned. If the 
user wishes to have the Image slightly reduced In size, thinning out of picture elements will be perfonmed. This way, 
the image size is re-adjusted. 

[0250] For performing the pk:ture element Interpolation, an area - other than the edge - where the density change Is 

40 small Is selected. Tlie same Is the case with the thinning out. 

[0251] Instead of the interpolation and thinning out, there is a technique in which the original image is transfomned 
in a frequency area and shortage components are added or excessive components are taken out from the high fre- 
quency. A proper technique is selected with the processing time, CPU capacity etc. taken Into consideration. 
[0252] EsEI output means 1214 outputs an enlarged image obtained by image fine-adjustment means 3704, that Is, 

45 to display the image on CRT etc., to print it out or refers it to other devtees. 

[0253] According to the present embodiment, as set forth above, a detailed enlarged image obtained is shown to 
the user who judges if the size and resolution are just right. After the user finds the Image a desired one, a series of 
enlargement steps can be suspended, and there is no need to set the enlargement ratio in advance and it is possible 
to enlarge an image simply to a size as desired by the user. 

50 

Embodiment 21 

[0254] Finally, there will be explained the Image processing device of Embodiment 21 . The present embodiment Is 
an invention related to effk:lency of estimating an enlarged Image of a color original Image. 
55 [0255] FIG. 38 is a block diagram showing the anrangement of an image processing device of this embodiment, and 
there will be described the operation of this image processing device. 

[0256] First, from a color original image inputted by image input means 10, SC selecting means 2800 selects the 
green component as standard color component. And TR depriving means 2801 finds simple ratio ratio_r of the red 
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component and ratio.b of the blue component to the green component. This process is the same as that in Embodiment 
1 6 and will not be explained. 

[0257] Then, as the image processing devices in Embodiments 17, 18 and 19 of the present invention, Standard 
Component Image (SCI) regulating means 3802 and Standard Image (SI) enlarging means 3803 enlarge the standard 
color component or the green component. And an enlarged image of the standard color thus obtained is subjected to 
picture element interpolation or thinning out by Standard Enlarged Image (El) regulating means 3804 so as to obtain 
a desired Image size Ln picture elements x Ln picture elements. Furthemnore, Shortage Components (ShC) enlarging 
means 3805 multiplies the enlarged green component by the simple ratio ratio_r, ratio_b, thereby preparing data on 
the remaining red and blue components. 

[0258] Enlarged Color Image (EC I) recomposing means 3806 combines these three enlarged components into one, 
thus producing an enlarged image of the color original Image. EsEI output means 1214 outputs an enlarged image 
obtained by image fine-adjustment means 3704, that is, to display the image on CFTT etc., to print it out or refers it to 
other devices. 

[0259] Through that process, it is possible to simplify and speed up the processing, eliminating need of enlarging 
each of a plurality of components making up a color original Image. 
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4001 inverse orthogonal transform means 

4002 Estimated Enlarged Image (EsEt) output means 



Claims 

1 . An Image processing device for processing Image data Inputted by image inputting means, said device comprising: 

Original Images orthogonal transfonning means for performing orthogonal transform on said image data to 
generate the frequency components of an original image, 

Low Frequency Components extracting means for extracting the low frequency components from said fre- 
quency components of the original image, 

High Frequency Components encoding means for finding and encoding relation infomnation between said low 
frequency and the remaining high frequency components of said frequency components of the original image, 
and 

Codes Synthesizing means for synthesizing said tow frequency components and said relation infomnation and 
generating simplified image data. 

2. The image processing device of claim 1 which comprises: 

Low Frequency Components compression means for compressing the data size of said low frequency com- 
ponents and 

wherein said Codes Synthesizing means synthesizes the tow frequency components compressed by 
said Low Frequency Components compression means and said relation information and generates simplified 
Image data. 

3. The image processing device of dalm 1 further comprising storage medium for storing said simplified image data. 

4. The image processing device of claim 1 further comprising: 

Reduced Images generating means for generating reduced images by perfomiing inverse orthogonal trans- 
form on said low frequency components corresponding to a reduction size specified in advance, and 
Reduced Images displaying means for displaying said reduced images. 

5. An image processing device which connprises: 

Low Frequency Components decoding means for extracting tow frequency components from the simplified 
image data obtained by synthesizing the low frequency components extracted from the frequency components 
of an original Image obtained by perfomiing orthogonal transfonn on the original image and the relation Infor- 
mation between said low frequency components and the remaining high frequency components, and 
High Frequency Components decoding means for taldng out said relation information from said simplified 
Image data and decoding the high frequency components on the basis of said low frequency components, 
and said relation infomnation and, 

Original Images outputting means for combining said low frequency components and said high frequency 
components, performing inverse orthogonal transform, and restoring and outputting the original Image. 

6. The Image processing device of claim 5 for taking out said simplified image data from the storage medium. 

7. An image processing device for processing image data inputted by image inputting means, said device comprising: 

Original Images orthogonal transforming means for performing orthogonal transform on said image data to 
generate the frequency component of an original image, 

Shortage Components estimating means for estimating from said frequency components of the original image 
the shortage frequency components when the original image is enlarged in accordance with a desired en- 
largement ratio, 

Basic Components extracting means for extracting frequency components - as basic components - necessary 
for restoring a basic image with the size specified in advance from said frequency components of the original 
image arid said estimated frequency components. 
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Multiple Image encoding means for finding and encoding the relation infomiation between said basic compo- 
nents and said estimated frequency components, and 

Multiple Codes Synthesizing means for synthesizing said basic components and said relation Infomnation and 
generating multiple simplified image data. 

5 

8. The image processing device of claim 7 which comprises: 

Basic Components compression means for compressing the data size of said basic components, and 
Codes Synthesizing means for synthesizing the basic components compressed by Basic Components com- 
10 pression means and said relation infonmation and generating the simplified image data. 

9. The image processing device of claim 7 which comprises a storage medium for storing said multiple simplified 
image data. 

IS 10. The image processing device of claim 7 which comprises: 

Basic Images generating means for generating basic images by perfonming inverse orthogonal transfonn on 

said basic components, and 

Basic Images displaying means for displaying said basic images. 

20 

11. An Image processing device which comprises: 

Basic Components decoding means forfaiting out basic components from multiple simplified image data gen- 
erated on the basis of basic components extracted from the frequency components of a plurality of images of 

25 enlarged sizes to restore a basic image with the size specified in advance and the relation infomiation between 

said basic components and the remaining high frequency components of the enlarged image, 
Object Frequency decoding means for taking out the relation Infomnatlon between said basic components and 
high frequency components of an enlarged image of a desired size from said multiple simplified image data 
and decoding said high frequency components on the basis of said basic components and said relation infor- 

30 mation, and 

Object Images outputting means for combining said basic components and said high frequency components, 
perfomiing inverse orthogonal transform, and restoring and outputting said enlarged Image of the desired size. 

12. The image processing device of claim 11 wherein said multiple simplified image data is taken out from said memory ' 
35 medium. 

13. The image processing device of daim 12 wherein the data starts to be taken out from said memory medium on 
the basis of an instruction signal from the user. 

40 1 4. An image processing device for processing image data Inputted by image inputting means, said device comprising: 

Inter-pbture element interpolating means for performing interpolation between picture elements on said image 
data In accordance with a desired enlargement ratio 

Convolution means for convoluting the interpolated enlarged image obtained by said inter-picture element 
45 interpolating means, 

Convergence judging means for judging if pre-set convergence conditions are satisfied on the basis of the 
difference between the mean values of convolution values of the enlarged image obtained at the k-th convo- 
lution and the enlarge image obtained at the (K - 1)-th convolution, and 

Enlarged Images output means for outputting the enlarged image obtained at the k-th convolution in case 
so Convergence judging means judges that the convergence conditions are satisfied. 

15. An image processing method whk:h comprises the steps of: 

low frequency component extracting for extracting low frequency components from the frequency components 
S5 of a specific image, 

high frequency component encoding for finding relation infomiation between said low frequency components 
and the remaining high frequency components, and 

simplified image synthesizing for synthesizing said low frequency components and said relation information 
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to generate simplified Image data. 

16. An image processing method which comprises the steps of: 

low frequency component extracting for extracting low frequency components from the frequency components 
of a specific image, 

high frequency component encoding for finding relation inf omrtation between said low frequency components 
and the remaining high frequency components, 

synthesizing for synthesizing said tow frequency components and said relation infomiation to generate sim- 
plified image data, 

decoding for decoding said low frequency components and said relation information from said simplified image 
data, and 

image outputting for combining said decoded low frequency components and said relation information, per- 
forming inverse orthogonal transform, then outputting said specific image. 

17. An image processing method which comprises a step of: 

basic component extracting for extracting low frequency components - as basic component - that can be 
regarded as common to the images of the respective sizes from the frequency components of an image of a 
specific size and enlarged images of said image of the specific size, 

multiple image encoding for finding the relation information between said basic component and the remaining 
high frequency components of the respective enlarged images, and 

multiple simplified image synthesizing for synthesizing said basic conrtponent and said relation informatbn to 
generate multiple simplified Image data. 

18. An image processing method which comprises the steps of: 

basic component extracting for extracting low frequency components - as basic component - that can be 
regarded as common to the Images of the respective sizes from the frequency components of an image of a 
specific size and a plurality of enlarged images of said image of the specific size, 

multiple encoding for finding the relation infonnation between said basic component and the remaining high 
frequency components of the respective enlarged Images, 

object frequency decoding for decoding said basic components and said relation Information from the multiple 
simplified image data obtained by synthesizing said basic components and said relation Information, and 
image outputting for enlarging said specific Image to a desired size by combining said decoded basic compo- 
nents and said relation information and perfomning inverse orthogonal transform and then outputting said 
enlarged Image. 

19. A recorded medium on which a program is recorded, said program extracting low frequency components from the 
frequency components of a specific image, finding the relation information between said low frequency components 
and the remaining high frequency components, synthesizing said low frequency components and said relation 
infomiation to generate simplified Image data. 

20. A recorded medium on which a program Is recorded, said program extracting low frequency components from the 
frequency components of a specific image, finding the relation infomiation between said low frequency components 
and the remaining high frequency components, generating simplified image data by synthesizing said low frequen- 
cy components and said relation information and at the same time decoding said low frequency components and 
said relation information from said simplified image data, combining said decoded low frequency components and 
said relation infonmation, and perfomiing orthogonal transfomi, thereby outputting said specific image. 

21. A recorded medium on which a program is recorded, said program extracting low frequency components - that 
can be regarded as common to the images of the respective sizes - as basic components fmm the frequency 
components of an image of a specific size and enlarged images of said image of the specific size, finding the 
relation infonnation between said basic components and the remaining high frequency components, and synthe- 
sizing said basic components and said relation information, thereby generating multiple simplified image data. 

22. A recorded medium on which a program is recorded, said program extracting low frequency components - that 
can be regarded as comnwn to images of the respective sizes - as basic components from an image of a specific 
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size and the frequency components of a plurality of enlarged images, finding the relation infomiation between said 
basic components and the remaining high frequency components of the respective enlarged images, and decoding 
said basic components and said relation infomiation from multiple simplified image data obtained by synthesizing 
said basic components and said relation infomiation, and at the same said combining decoded basic components 
and said relation infomiation and enlarging said specific image to a desired size by perfomning inverse orthogonal 
transform, and then outputting said specific image. 

23. An image processing device for acquiring an enlarged image by enlarging a specific image data to a desired 
enlargement size, said device comprising: 

Original Images orthogonal transforming means for generating the frequency components of an original Image 
by perfomning orthogonal transform on said specific image data, 

Enlarged Frequency estimating means for estimating the frequency components of said enlarged image by 

perfomning nonlinear interpolation on said original image frequency components, and 

Inverse orthogonal transform means for acquiring an enlarged image data by perfomning Inverse orthogonal 

transform corresponding to said enlargement size on said estimated frequency components of the enlarged 

image. 

24. An image prbcessing device for acquiring an enlarged image by enlarging a specific image data to a desired 
enlargement size, said device connprising: 

Original Images orthogonal transforming means for generating the frequency components of an original image 
by performing orthogonal transform on said specific image data, 

Edge Generating means for generating edge image data of said specific image data by performing a specific 
edge detection on said specific image data, 

Enlarged Edge estimating means for estimating the frequency components of an enlarged edge image on the 
basis of said edge image data, 

Low Frequency Component substituting means for generating substituted frequency components by substi- 
tuting the low frequency area of the frequency components of said estimated enlarged edge image with said 
original image frequency components and 

Inverse orthogonal transfomn means for acquiring said enlarged image by perfomning inverse orthogonal trans- 
form on said substituted frequency components. 

25. The image processing device of claim 24 wherein said Enlarged Edge estimating means is provided with: 

Enlarged Edge approximating means for estimating the edge image data of said enlarged image by perfomning 
linear approximation on said edge Innage data, and 

Edge Frequency generating means for finding the frequency components of said edge image data of said 
enlarged image by orthogonal transfomn. 

26. The image processing device of claim 24 wherein said Enlarged Edge estimating means is provided with: 

Edge Images orthogonal transfomning means for generating the frequency cornponents of said edge image 
data by orthogonal transform and 

Edge Frequency estimating means for estimating the frequency components of edge Image data after en- 
largement on the basis of the frequency components of said generated edge Image data. 

27. The image processing device of claim 25 wherein said Edge Frequency generating means estimates the frequency 
components of the edge image data after enlargement by perfonnlng linear Interpolation on the frequency com- 
ponents of the edge image data obtained by said Original Images orthogonal transfomning means. 

28. The Image processing device of claim 25 wherein said Edge Frequency estimating means estimates the frequency 
components of the edge image data after enlargement by applying a nonlinear interpolation technique to the fre- 
quency components of the edge image data obtained by Original Images orthogonal transfomning means. 

29. An image processing device for acquiring an enlarged image by enlarging a specific image data to a desired 
enlargement size, said device comprising: 
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Block dividing means for dividing said specific image data into a plurality of overlapped blocks, 

Block Images Orthogonal transfomn means for generating the frequency components of images within the 

blocks by perfonning orthogonal transform on the images within the blocks obtained by said block dividing 

means, 

Enlarged Block Images frequency estimating means for estimating the frequency components of enlarged 
images of the images within the blocks on the basis of the frequency components of said images within the 
blocks, 

Block frequency extracting means for extracting frequency components within the blocks from said estimated 
frequency components, said frequency components being necessary for said enlarged image, 
Block Inverse orthogonal transfomn means for acquiring an enlarged block image by perfomiing inverse or- 
thogonal transform-corresponding to the size of said frequency components within the blocks-on said frequen- 
cy components within the blocks, and 

Enlarged Images recomposing means for acquiring said enlarged image by combing a plurality of said enlarged 
blocks. 

30. An image processing device for acquiring an enlarged image by enlarging a specific image data to a desired 
enlargement size, said devk:e comprising: 

Block dividing means for dividing said specify image data into a plurality of not overiapped blocks, 

Block data transfomriing means for transfomning said image data within the blocks on the basis of a specif k: 

transfomn function so that specific border conditions are satisfied on the border of said blocks, 

Block Images Orthogonal transfomn means for generating the frequency components of the images within the 

blocks by perfomiing orthogonal transform on said transfomned image data within the blocks. 

Enlarged Block Images frequency estimating means for estimating the frequency components of enlarged 

images of the images within the blocks on the basis of the frequency components of said images within the 

blocks, 

Block inverse orthogonal transform means for acquiring an enlarged block Image by performing orthogonal 
transfonm - corresponding to the frequency component size of said enlarged images - on the frequency com- 
ponents of said estimated enlarged images, and 

Enlarged Images recomposing means for acquiring said enlarged image by combining a plurality of said en- 
larged block innages. 

31. An image processing devtee for acquiring an enlarged image by enlarging a specific image data to a desired 
enlargement size, said device comprising: 

Shortage Component enlarging means component for selecting a standard from among the color components 
making up said color image, 

Transfonm Ratio deriving means for deriving the transfomn ratio whbh is used when data on the other cok>r 
components Is derived from saki basK component data. 

Standard Image orthogonal transforming means for generating the frequency component of the basic conn- 
ponent by perfomning orthogonal transform on said basic component data. 

Standard Enlarged Frequency estimating means for estimating the frequency components of an enlarged 
Image from the frequency components of said standard component, 

Standard Inverse orthogonal transforming means for acquiring the enlarged image data of the standard conn- 
ponent by perfomning Inverse orthogonal transfomn - corresponding to the frequency component size of said 
enlarged image - on the frequency component of the enlarged image of said standard component. 
Shortage Component enlarging means for estimating the other color component data in said enlarged Image 
on the basis of the enlarged image data of said standard component and said transfomi ratio, 
Enlarged Color Image recomposing means for generating said enlarged image by synthesizing the enlarged 
image data of said standard component and said other color component data. 

32. An image processing method for acquiring an enlarged image by enlarging a specifk: image data to a desired 
enlargement size, said method comprising the steps of: 

original image orthogonal transforming for performing orthogonal transfomn on said specifk: image data to 
generate the frequency components of the original image, 

enlarged frequency estimating for estimating the frequency components of said enlarged image by perfomning 
nonlinear interpolation on the frequency component of said original image, and 
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Inverse orthogonal transforming for perfonning inverse orthogonal transfomn - con^esponding to said enlarged 
size • on the frequency components of said estimated enlarged image to acquire enlarged image data. 

33. An image processing method for acquiring an enlarged image by enlarging a specific image data to a desired 
enlargement size, said method comprising the steps of: 

original image orthogonal transforming for perfonning orthogonal transform on said specific hmage data to 
generate the frequency components of an original image, 

edge generating for generating the edge image data of said specific data by a specific method, 

enlarged edge estimating for estimating the edge image data of said enlarged image by performing linear 

approximation on said edge image data, 

edge image frequency generating for finding the frequency component of the edge image data of sard enlarged 
image by orthogonal transforming step, 

low frequency substituting for generating a substituted frequency component by substituting the low frequency 
area of the frequency component of said edge image data with said original image frequency component, and 
inverse orthogonal transfomning for inverse orthogonal transfonming said substituted frequency component. 

34. An image processing method for acquiring an enlarged image by enlarging a specific image data to a desired 
enlargement size, said method comprising the steps of: 

original image orthogonal transforming for performing orthogonal transfomn on said specific image data to 
generate an original image frequency component, 

edge generating for generating the edge image data of said specific image data by a specific method, 
edge image orthogonal transfomning for generating the frequency component of said edge image data by 
orthogonal transfomning, 

edge frequency estimating for estimating the frequency component area of the edge image after enlargement 
on the basis of the frequency components of said generated edge image, and 

low frequency substituting for substituting the low frequency area of the frequency component of said edge 
Image after enlargement with said original image frequency component. 

35. The image processing method of claim 34 wherein said edge frequency estimating step estimates the frequency 
component of the edge image data after enlargement by perfonning linear approximation on the frequency com- 
ponent of the edge innage data obtained In said original image orthogonal transfonning. 

36. The image processing method of claim 34 wherein said edge frequency estimating step estimates the frequency 
component of the edge image data after enlargement by applying a nonlinear interpolation technique on the fre- 
quency component of the edge image data obtained In said original image orthogonal transforming step. 

37. An image processing method for acquiring an enlarged Image by enlarging a specific image data to a desired 
enlargement size, said method comprising the steps of: 

block dividing for dividing said specific Image data into a plurality of overiapped blocks, 
Image orthogonal transfomning for generating the frequency components of the images within the blocks by 
performing orthogonal transform on the images within said blocks obtained in said block dividing step, 
enlarged block image frequency estimating for estimating the frequency components of the enlarged images 
of the images within the blocks on the basis of the frequency components of the images within the blocks, 
in-block frequency extracting for extracting the frequency component within the block - necessary for said 
enlarged images - from said estimated frequency components, 

block orthogonal transfomning for acquiring enlarged block images by perfonning inverse orthogonal transfomn 
- con^esponding to the sizes of the frequency components within said blocks - on said frequency components 
with the blocks, and 

enlarged image composing for acquiring said enlarged images by combing a plurality of said enlarged block 
Images. 

38. An image processing method for acquiring an enlarged image by enlarging a specif k; image data to a desired 
enlargement size, said method comprising the steps of: 

block diving for dividing said specifk^ image data into a plurality of not overiapped blocks. 
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in-block data transforming for transforming the image data within said bloclcs on the basis of a specific transfonm 

function so that specific border conditions are satisfied on the border of said blocks, 

block image orthogonal transforming for generating the frequency components of the Images within the bk>cks 

by performing orthogonal transform on said transformed Image data within the blocks, 

enlarged block image frequency estimating for estimating the frequency components of the enlarged images 

within the blocks on the basis of the frequency components of the images within said blocks, 

block inverse orthogonal transfomriihg for acquiring enlarged images by perfonning Inverse orthogonal trans- 

f orni - corresponding to the frequency component sizes of said enlarged Images - on the frequency components 

of said estimated enlarged images, and 

Image composing for acquiring sajd enlarged images by combining a plurality of said enlarged block images. 

39. An image processing method for acquiring an enlarged image by enlarging a specify Image data to a desired 
enlargement size, said method comprising the steps of: 

standard component selecting for selecting a standard color component within the cok)r components making 
up said specific color image, 

transform ratio deriving for deriving the transfomn ratio whk:h is used when the other color component data 
are derived from said standard component data, 

standard image orthogonal transforming for generating the frequency component of the standard component 
by perfomning orthogonal transform on the data of said bask: component, 

standard enlarged frequency estimating for estimating the frequency components of the enlarged images from 
the frequency conrtponent of said standard component, 

standard inverse orthogonal transform for acquiring the enlarged image data of the standard component by 
perfomning Inverse orthogonal transfomn - corresponding to the size of the frequency component of said en- 
larged image- on the frequency component of the enlarged image of said standard component, 
shortage component enlarging for estimating the other color component data in said enlarged image on the 
basts of the enlarged Image data of said standard component and said transfbnming ratio, and 
enlarged color image re-composing for generating said enlarged image by synthesizing the enlarged image 
data of said standard component and said other color component data. 

40. A recorded medium on which a program is recorded, said program - when obtaining an enlarged image by enlarging 
a specific image data to a desired enlargement size - generating the frequency component of an original image 
by perfonning orthogonal transform on said specific image data, estimating the frequency component of said en- 
larged image by pertonming nonlinear interpolation on said original image frequency component, and performing 
Inverse orthogonal transfomn corresponding to said enlarged size on the frequency component of said estimated 
enlarged image, thereby acquiring said enlarged image. 

41 . A recorded medium on which a program is recorded, said program - when obtaining an enlarged innage by enlarging 
a specific image data to a desired enlargement size - generating the frequency component of an original Image 
by perfonning orthogonal transform on said specific image data and at the same time generating the edge image 
data of said specific image data by a specific method, estimating the enlarged image by perfonning linear approx- 
imation on said edge image data, finding the frequency component of the edge image data of said enlarged image 
by orthogonal transfomn, generating a substituted frequency component by sut>stituting the low frequency area of 
said frequency component the enlarged edge image with said original image frequency component, and perfomning 
inverse orthogonal transform on said substituted frequency component, thereby acquiring said enlarged image. 

42. A recorded medium on which a program is recorded, said program - when obtaining an enlarged image by enlarging 
a specifb image data to a desired enlargement size - generating the frequency component of the original image 
by perfonning orthogonal transfomn on said specific image data and at the same time generating the edge image 
data of said specify Image data by a specific method, generating the frequency component of said edge image 
data by orthogonal transfomn, estimating the frequency area of the edge image after enlargement on the basis of 
the frequency component of said edge image, and substituting the low frequency area of the frequency component 
of said edge image after enlargement with said original image frequency component. 

43. A recorded medium on which a program is recorded, said program - when obtaining an enlarged image by enlarging 
a specific image data to a desired enlargement size - dividing the specific image data into a plurality of overiapped 

. blocks, generating the frequency components of the images within the blocks by perfonning orthogonal transform 
on the images within the blocks thus obtained, estimating the frequency components of enlarged Images within 
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the blocks on the basis of the frequency components of the images within said blocks and at the same time ex- 
tracting thefrequency components within blocks necessary forsald enlarged Images from said estimated frequency 
components, acquiring enlarged block images by perfomiing inverse orthogonal transfomn-corresponding to the 
sizes of the frequency components within said blocks - on said frequency components within the blocks, and 
combing a plurality of said enlarged block images, thereby acquiring said enlarged image. 

44. A recorded medium on which a program is recorded, said program - when obtaining an enlarged image by enlarging 
a specific image data to a desired enlargement size - dividing the specific image data into a plurality of not over- 
tapped blocks, transfomning the image data within said blocks on the basis of a specific transfonm function so that 
specific border conditions are satisfied on the border of said block, generating the frequency components of images 
within the blocks by performing orthogonal transform on said transformed image data within the blocks, and at the 
same time estimating the frequency components of the enlarged images of the images within the blocks on the 
basis of the frequency components within said images within the blocks, acquiring enlarged block images by per- 
fomiing inverse orthogonal transfonn - corresponding to the frequency component size of said enlarged images - 
on said enlarged image frequency components, and combining a plurality of said enlarged block Images, thereby 
obtaining said enlarged image. 

45. A recorded medium on which a program is recorded, said program - when obtaining an enlarged image by enlarging 
a specifk: image data to a desired enlargement size - selecting a standard color component from among the color 
components making up said specific color image, deriving a transform ratio which is used when the other color 
component data are derived from said standard component data, generating the frequency component of the 
standard component by perfomning orthogonal transform on said standard component data, estimating the fre-' 
quency component of the enlarged image from the frequency component of said standard component and at the 
same time acquiring the enlarged image data of the standard component by performing inverse orthogonal trans- 
fonn - corresponding to the frequency component size of said enlarged Image - on the frequency components of 
said standard component, and estimating the other color component data in said enlarged image on the basis of 
the enlarged image data of said standard component and said transfomr) ratio, thereby obtaining said enlarged 
Image. 

46. An image processing device for acquiring an enlarged image of Ln picture elements x Lm picture elements by 
enlarging an original image of n pk^ure elements x m picture elements, said device comprising: 

Inputted Images regulating means for interpolating or thinning out (hereinafter both expressed as regulate) 
said original image to Ln/2 pic^re elements x Lm/2 picture elements and 

Image enlarging means for generating an enlarged image by applying an enlargement technique based on 
Wavelet transfonn on the image regulated by Inputted Images regulating means . 

47. An image processing devk:e for acquiring an enlarged image by enlarging an original Image, said device compris- 
ing: 

Enlarging Process initializing means for setting the original image as an enlargement object image, 
Enlarging Object Images means for generating an enlarged image having four tinnes as many pk;ture elements 
by applying an enlargement technique based on Wavelet transfonn to said enlargement object Image, 
Multiple Processing ending judge means for setting said enlarged image - obtained from Enlarging Object 
Images means - as enlargement object image and retums the process to Enlarging Object Images means, 
Enlarged Images presenting means for visually presenting an enlarged image obtained from Enlarging Object 
Images means. 

Image fine-adjustment means for enlarging or reducing the enlarged innage presented by said Enlarged Innages 
presenting means, 

Enlarged Images output means for outputting an image obtained from said image fine-adjustment means. 

48. The image processing device of claim 46 or claim 47 wherein said Enlarging Object Images means comprising: 

Edge generating means for generating edge images - in a plurality of specific directions - of an image regulated 
by said Inputted Images regulating means , and 

Leveling up means for generating an image having four times as many pk:ture elements by regarding said 
respective edge images and said regulated original image as sub-band components in Wavelet transfonn and 
by performing inverse Wavelet transform on said sub-band components. 
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49. The image processing device of claim 48 wherein a plurality of said directions are three directions - vertical, hor- 
izontal and oblique. 

50. The image processing device of claim 46 or claim 47 wherein said Enlarging Object Images intializing means 
comprising: 

Input fine-adjustment means for re-regulating the riumbers of the picture elements of said regulated original 
image in the horizontal direction and the vertical direction to multiples of 2, 

Leveling down means for generating a transformed image by performing Wavelet transfomn on said re-regu- 
lated original image, 

Reference Components generating means for generating edge images - in a plurality of said specific directions 
- from a plurality of sub-band components situated in the of said transformed image, 
Edge generating means for generating edge Image - In plurality of specific direction - of image re-regulated 
be said Input fine-adjustment means, 

Correction estimating means for finding the relation between said respective edge images and the sub-band 
components belonging to the low frequency space corresponding to said specific directions of said transformed 
images, 

Component estimating means for estimating the respective sub-band components in Wavelet transform by 
correcting said respective edge image of re-regulated original image according to the results of said correction 
estimating means, and 

Leveling up means for generating an enlarged image having fourtimes as many picture elements by perfomnlng 
Inverse Wavelet transform on said respective sub-band components and said re-regulated original Image. 

51 . The image processing device of claim 50 wherein Reference Components generating means finds said edge Image 
by Lapladan filter from the sub-band components situated in the low frequency area of said transformed Image. 

52. The image processing device of daim 46 or claim 47 wherein in case said Ln/2 picture elements and Lm/2 picture 
elements are integers, an inputted original image is regulated by said Inputted Images regulating means, and from 
the regulated image, said Enlarging Object Images means generates an enlarged image, and wherein in case at 
least one of Ln/2 picture elements and Lm/2 picture elements Is not an integer, the inputted original image is 
regulated by said Inputted Images regulating means , and from the regulated Image, said Enlarging Object Images 
means generates an enlarged image, and then the enlarged image obtained by Enlarging Object Images means 
is regulated to Ln picture elements x Lm picture elements by means for regulating enlarged image. 

53. An Image processing device for acquiring an enlarged image of Ln picture elements x Lm picture elements by 
enlarging a color Image of n picture elements x m picture elements, said device comprising: 

Standard Component selecting means for selecting a standard color component from among the components 
making up said color image, 

Transfomri Ratio deriving means for deriving the transfomn ratio which is used when deriving the other color 
components from the standard color selected by said Standard Component selecting means, 
Standard Component Image regulating means for regulating the standard color component of the Inputted 
original image to Ln^ picture elements x Lm/2 picture elements. 

Standard Image enlai-ging means for generating a standard enlarged image by applying a method based on 
Wavelet transform to said regulated standard color component. 

Standard Enlarged Image regulating means for regulating said standard enlarged image to a desired size Ln 
picture elements x Lm picture elements, 

Shortage Component enlarging meanss for estimating the enlarged images of the other color components by 
applying said transfonm ratio to said regulated standard enlarged image, and 

Enlarged Color Image recomposing means for generating said enlarged image by synthesizing said standard 
enlarged image and said enlarged images of the other color components. 

54. The image processing device of claim 50 wherein said conrection estimating means finds the difference image - 
as respective correction component image - between the edge images in said respective directions and the sub- 
band components belonging to the low frequency area corresponding to said spedf ic directions of said transfomned 
Images, and wherein said component estimating means enlarges said respective connection component Images 
to images with four times as many picture elements by liner interpolation, and adds the corresponding edge images 
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to said correction component images. 

55. An image processing method for acquiring an enlarged Image of Ln picture elements x Lm picture elements by 
enlarging an original image of n picture elements x m picture elements, said method comprising the steps of: 

Inputted Image regulating for regulating said original image to Ln/2 picture elements x Lnn/2 picture elements, 
and 

image enlargement for generating an enlarged image by applying an enlargement method based on Wavelet 
transform to the image regulated by Inputted Images regulating means . 

56. An image processing method for acquiring an enlarged image by enlarging an original Image, said method com- 
prising the steps of: 



enlargement Initializing for initializing the enlargement process by setting the original image as enlargement 
object Image, 

object image enlarging for generating an enlarged image having four times as many picture elements by ap- 
plying an enlargement method based on Wavelet transfomi to said enlargement object Image, 
multiple proceeding end judging for setting - as enlargement object image - the enlarged image obtained by 
said Enlarging Object Images means and retums the process to said Enlarging Object Images means 
enlarged image presenting for visually presenting the enlarged Image obtained by said Enlarging Object Im- 
ages means, 

Image regulating for enlarging or reducing the enlarged image presented by said Enlarging Object Images 
means, and 

enlarged image outputting for outputting the image obtained from said image fine-adjustment means. 

57. A recorded medium on which a program is recorded, said program - when acquiring an enlarged image of Ln 
picture elements x Lm picture elements by enlarging a color image of n picture elements x m picture elements - 
regulating said original image to Ln/2 picture elements x Lm/2 picture elements, and applying an enlargement 
method based on Wavelet transfonn to the regulated image, thereby generating an enlarged Image. 

58. A recorded medium on which a program is recorded, said program - when acquiring an enlarged Image by regu- 
lating an original image - setting the original image as enlargement object image, and generating an enlarged 
image having fourtlmes as many picture elements by applying an enlargement method based on Wavelet transform 
to said enlargement object image, setting the enlarged image as enlargement object Image and returning the 
process to Enlarging Object Images means and at the same time visually presenting the enlarged Image, thereby 
enlarging or reducing said enlarged image. 
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